Problem/Motivation

\Drupal\Core\Database\Connection::query() and ::defaultOptions() refer to a 'target' option where the database connection can be changed. This doesn't actually work, as with the current API you first define the target when setting up the connection object, then execute operations on it - you can't change it later. This is was presumably copied as-is from the db_*() functions which did support the 'target' option but now should be removed from the documentation.

Proposed resolution

  1. Remove the references to 'target' in the Connection class, code and documentation.
  2. Ensure that legacy db_* wrappers manage the connection before invoking the OO API, and that the 'target' key is not passing through
  3. As a safety measure to prevent regression (some wrong usages were fixed in #3001216: Use the database.replica service where appropriate), and to help contrib to converge, raise a deprecation message in case a 'target' key is landing in Connection::query for any reason

Remaining tasks

Review patch

User interface changes

None

API changes

None

Data model changes

None

CommentFileSizeAuthor
#67 interdiff_66-67.txt616 bytesmondrake
#67 3000931-67.patch12.98 KBmondrake
#66 3000931-66.patch12.99 KBmondrake
#66 interdiff_61-66.txt5.53 KBmondrake
#61 interdiff-3000931-58-61.txt516 bytesvoleger
#61 3000931-61.patch12.9 KBvoleger
#58 interdiff-3000931-56-58.txt9.05 KBvoleger
#58 3000931-58.patch13.03 KBvoleger
#56 interdiff-3000931-53-56.txt8.89 KBvoleger
#56 3000931-56.patch13.41 KBvoleger
#53 3000931-53.patch12.81 KBmondrake
#53 interdiff_50-53.txt1.1 KBmondrake
#50 3000931-50.patch12.27 KBmondrake
#50 interdiff_46-50.txt3.55 KBmondrake
#46 interdiff_43-46.txt2.24 KBmondrake
#46 3000931-46.patch12.24 KBmondrake
#43 3000931-43.patch12.36 KBmondrake
#43 interdiff_40-43.txt1.23 KBmondrake
#40 interdiff_26-40.txt2.36 KBmondrake
#40 3000931-40.patch12.36 KBmondrake
#32 interdiff-3000931-30-32.txt463 bytesvoleger
#32 3000931-32.patch20.12 KBvoleger
#30 interdiff-3000931-26-30.txt6.86 KBvoleger
#30 3000931-30.patch20.12 KBvoleger
#26 3000931-26.patch13.54 KBmondrake
#26 interdiff_22-26.txt2.04 KBmondrake
#22 interdiff_20-21.txt1.26 KBmondrake
#22 3000931-21.patch12.67 KBmondrake
#20 interdiff_14-20.txt687 bytesmondrake
#20 3000931-20.patch12.76 KBmondrake
#18 3000931-alternative-18.patch3.05 KBlongwave
#14 3000931-14.patch12.08 KBandypost
#14 interdiff.txt2.35 KBandypost
#11 3000931-11.patch12.03 KBmondrake
#11 interdiff_9-11.txt2.11 KBmondrake
#9 interdiff_5-7.txt9.01 KBmondrake
#9 3000931-7.patch10.97 KBmondrake
#5 3000931-5.patch1.96 KBmondrake
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

longwave created an issue. See original summary.

catch’s picture

Issue tags: +Novice
mondrake’s picture

I guess it should also be removed from Connection::defaultOptions.

Shall we also add a conditional deprecation that triggers if something is still eventually passing the 'trigger' key to the $options array of Connection::query?

mondrake’s picture

Assigned: Unassigned » mondrake

On it

mondrake’s picture

Status: Active » Needs review
FileSize
1.96 KB

Initial patch, will fail with deprecations from testing legacy db_* functions.

mondrake’s picture

Assigned: mondrake » Unassigned
mondrake’s picture

Component: documentation » database system

Status: Needs review » Needs work

The last submitted patch, 5: 3000931-5.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
FileSize
10.97 KB
9.01 KB

Now changing the deprecated db_* functions to remove the $options['target'] key before invoking OO API.

Status: Needs review » Needs work

The last submitted patch, 9: 3000931-7.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
Issue tags: -Novice
FileSize
2.11 KB
12.03 KB

I cannot understand the failures in Search tests here. Will investigate further, any pointer appreciated.

For the moment, added a deprecation tests for the deprecation of the 'target' key usage in Connection::query.

Status: Needs review » Needs work

The last submitted patch, 11: 3000931-11.patch, failed testing. View results

longwave’s picture

I don't think a deprecation is the right thing here. This has never worked in D8 when called via OO, only the db_* methods explicitly supported it.

If we do want to throw errors how about checking that 'target' is either unset or matches the connection object target and throwing an error if it is something else, as this is likely indicating a bug in the caller - they are not talking to to the database they expect?

andypost’s picture

Status: Needs work » Needs review
Issue tags: +Needs issue summary update
FileSize
2.35 KB
12.08 KB

Added link to issue with discussion and fixed linting

- As db_ wrappers still need this option we should unset it and continue select target through option
- throwing error still needed to notify about useless/non-working option passed and legacy needs clean-up (very helpful to catch bugs)

Great discussion and tl'dr from @catch makes sense to move to this issue summary and update links if needed, IMO no CR needed

Status: Needs review » Needs work

The last submitted patch, 14: 3000931-14.patch, failed testing. View results

andypost’s picture

Failed tests shows that we need to finish conversion of db_* leftovers first

mondrake’s picture

#14 thanks for fixing my c/p error :)

#16 well tests should not fail since we are removing the 'target' key in all db_* functions before calling the OO code in this patch already. I can not understand where the key is still leaking in from.

longwave’s picture

Status: Needs work » Needs review
FileSize
3.05 KB

I was thinking we could do something simpler, like this?

Status: Needs review » Needs work

The last submitted patch, 18: 3000931-alternative-18.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
FileSize
12.76 KB
687 bytes

#18 thanks @longwave. I think that will lead to WSOD on sites that are using modules still passing the target key, no? Deprecation is less intrusive as it will only fail in testing.

Your patch actually may be giving hints as to why #14 is failing. Here's a new patch following on #14.

longwave’s picture

I think that will lead to WSOD on sites that are using modules still passing the target key, no?

Yes - but only if the specified target is not actually the current connection target, and in that case, isn't this a bug on the caller's side? The caller is requesting a specific target, but they aren't actually using the target they requested

This is why the db_* calls still work with my patch, because they pass 'default' as the target, but that is OK when the 'default' connection is already selected.

mondrake’s picture

#20 does not make sense, it's removing the key just before invoking the check that it is there. See this one.

The last submitted patch, 20: 3000931-20.patch, failed testing. View results

andypost’s picture

I bet some code using to pass options[target] to injected database that's why it fails

mondrake’s picture

Found a couple of possible culprits. Patch on the way.

mondrake’s picture

CommentStatistics and Plugin/Search/NodeSearch are passing 'target' => 'replica' directly to OO methods.

longwave’s picture

  1. +++ b/core/modules/comment/src/CommentStatistics.php
    @@ -64,8 +64,7 @@ public function __construct(Connection $database, AccountInterface $current_user
    -    $options = $accurate ? [] : ['target' => 'replica'];
    -    $stats = $this->database->select('comment_entity_statistics', 'ces', $options)
    +    $stats = $this->database->select('comment_entity_statistics', 'ces')
    

    So this is actually a bug and it needs to choose a different connection based on $accurate.

  2. +++ b/core/modules/node/src/Plugin/Search/NodeSearch.php
    @@ -237,7 +237,7 @@ protected function findResults() {
         $query = $this->database
    -      ->select('search_index', 'i', ['target' => 'replica'])
    +      ->select('search_index', 'i')
    

    Similarly here we need a different database injected.

catch’s picture

We can't solve that with a different database connection though, because even when replica is specified it needs to fall back to default.

So this either means changing these places to use Database::getConnection() or adding an injectable service (that would just delegate to Database::getConnection() at least for now.

longwave’s picture

I do wonder if the fact that this has been around since 8.0 yet no-one has noticed so far means that no-one actually uses the 'replica' feature in production? I assume this is meant to take load off the primary for heavy read-only queries in a primary/replica setup.

voleger’s picture

How about the approach like this? Any patch updates are welcome.

Status: Needs review » Needs work

The last submitted patch, 30: 3000931-30.patch, failed testing. View results

voleger’s picture

Status: Needs work » Needs review
FileSize
20.12 KB
463 bytes

Status: Needs review » Needs work

The last submitted patch, 32: 3000931-32.patch, failed testing. View results

mondrake’s picture

I would suggest to do #30 in a separate issue (it's a prerequisite), and postpone this on that one. This one would prevent further bugs.

voleger’s picture

Status: Needs work » Postponed
Related issues: +#3001216: Use the database.replica service where appropriate

Filled #3001216: Use the database.replica service where appropriate and postponing on it this issue.

catch’s picture

#29 seems likely, I think it's a more common pattern at this point to shift listing queries to Solr (which will be faster than a query run against a replica would be). I have worked on one site that was running some read queries against replica, but it was on 6.x so couldn't actually use this feature anyway.

andypost’s picture

+++ b/core/modules/comment/src/CommentStatistics.php
@@ -52,20 +67,24 @@ class CommentStatistics implements CommentStatisticsInterface {
+    $this->connectionFactory = $connection_factory;
     $this->database = $database;
...
+    $this->replicaDatabase = $this->connectionFactory->getDatabaseConnection('replica');

main question here is do we need to make connection in constructor or not! and any reson to store connections when we have factory injected?

longwave’s picture

Status: Postponed » Active

The blocker is in so this can be worked on again, if there is even anything left to do here.

mondrake’s picture

Assigned: Unassigned » mondrake

We definitely still need to

  1. remove the wrong documentation
  2. adjust the legacy db_* functions to use database.replica where needed
  3. unset the 'target' key before invoking the OO API
  4. and I'd still keep the @trigger_error when query() is invoked with a 'target' key in $options, it would be useful to prevent regression and help contrib to comply

I will make a reroll from #26.

EDIT - point 2 is unnecessary as in database.inc we are using static calls and not the service.

mondrake’s picture

Status: Needs review » Needs work

The last submitted patch, 40: 3000931-40.patch, failed testing. View results

andypost’s picture

Re-queued

+++ b/core/tests/Drupal/KernelTests/Core/Database/DatabaseLegacyTest.php
@@ -431,7 +431,7 @@ public function testDbTruncate() {
-   * @expectedDeprecation Passing a 'target' key to Connection::query $options argument is deprecated in Drupal 8.0.x and will be removed before Drupal 9.0.0. Instead, get the target connection via Database::getConnection() passing the target in input and execute query() on it. See https://www.drupal.org/node/3000931
+   * @expectedDeprecation Passing a 'target' key to Connection::query $options argument is deprecated in Drupal 8.0.x and will be removed before Drupal 9.0.0. Instead, get the target connection via Database::getConnection() passing the target in input and execute query() on it. https://www.drupal.org/node/2993033

"See" is gone for some reason?

mondrake’s picture

Status: Needs work » Needs review
FileSize
1.23 KB
12.36 KB

"See" is gone for some reason?

Yes, effects of carpal tunnel syndrome on copy/paste operations :)

Thanks, this should do.

longwave’s picture

Instead, get the target connection via Database::getConnection() passing the target in input and execute query() on it

This doesn't scan well for me. What about something like "Instead, get the target connection directly via Database::getConnection() and then execute query() on it"?

Or even just "Instead, use Database::getConnection($target)->query()"?

The last submitted patch, 40: 3000931-40.patch, failed testing. View results

mondrake’s picture

Issue summary: View changes
Issue tags: -Needs issue summary update

Updated IS

andypost’s picture

RTBC +1 just minor nit

+++ b/core/lib/Drupal/Core/Database/Connection.php
@@ -600,6 +593,9 @@ protected function filterComment($comment = '') {
+      @trigger_error('Passing a \'target\' key to Connection::query $options argument is deprecated in Drupal 8.0.x and will be removed before Drupal 9.0.0. Instead, use Database::getConnection($target)->query(). See https://www.drupal.org/node/2993033', E_USER_DEPRECATED);

+++ b/core/tests/Drupal/KernelTests/Core/Database/DatabaseLegacyTest.php
@@ -428,6 +428,15 @@ public function testDbTruncate() {
+   * Tests deprecation of the $options 'target' key in Connection::query.
...
+   * @expectedDeprecation Passing a 'target' key to Connection::query $options argument is deprecated in Drupal 8.0.x and will be removed before Drupal 9.0.0. Instead, use Database::getConnection($target)->query(). See https://www.drupal.org/node/2993033

should we use full namespace for connetction & `query()`?

longwave’s picture

  1. +++ b/core/includes/database.inc
    @@ -473,10 +454,9 @@ function db_driver() {
    -  if (empty($options['target'])) {
    -    $options['target'] = NULL;
    -  }
    -  Database::closeConnection($options['target']);
    +  $target = empty($options['target']) ? 'default' : $options['target'];
    +  unset($options['target']);
    +  Database::closeConnection($target);
    

    Not sure this needs changing, db_close() has always been weird in that it takes an array and only ever uses one value.

  2. +++ b/core/includes/database.inc
    @@ -329,12 +313,9 @@ function db_select($table, $alias = NULL, array $options = []) {
    +  $target = empty($options['target']) ? 'default' : $options['target'];
    +  unset($options['target']);
    +  return Database::getConnection($target)->startTransaction($name);
    

    Similarly, not sure we need the unset here as $options is not actually used.

mondrake’s picture

voleger’s picture

Status: Needs review » Reviewed & tested by the community

Looks good

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/includes/database.inc
@@ -93,11 +91,9 @@ function db_query($query, array $args = [], array $options = []) {
+  return Database::getConnection($target)->queryRange($query, $from, $count, $args, $options);

@@ -130,11 +126,9 @@ function db_query_range($query, $from, $count, array $args = [], array $options
+  return Database::getConnection($target)->queryTemporary($query, $args, $options);

@@ -158,10 +152,9 @@ function db_query_temporary($query, array $args = [], array $options = []) {
+  return Database::getConnection($target)->insert($table, $options);

@@ -185,12 +178,9 @@ function db_insert($table, array $options = []) {
+  return Database::getConnection($target)->merge($table, $options);

@@ -214,10 +204,9 @@ function db_merge($table, array $options = []) {
+  return Database::getConnection($target)->update($table, $options);

@@ -241,12 +230,9 @@ function db_update($table, array $options = []) {
+  return Database::getConnection($target)->delete($table, $options);

@@ -270,10 +256,9 @@ function db_delete($table, array $options = []) {
+  return Database::getConnection($target)->truncate($table, $options);

@@ -301,10 +286,9 @@ function db_truncate($table, array $options = []) {
+  return Database::getConnection($target)->select($table, $alias, $options);

I think \Drupal\Core\Database\Connection::*() needs to deprecate the target option too. Atm we've only done ::query() but all the other methods should have this. Potentially this deprecation could live in \Drupal\Core\Database\Query\Query::__construct() - apart from for ::queryTemporary and ::queryRange but they both call query() so they're covered already.

Also the $options documentation foreach of them is a bit off. For example ::insert() says

   * @param array $options
   *   (optional) An associative array of options to control how the query is
   *   run. The given options will be merged with
   *   \Drupal\Core\Database\Connection::defaultOptions().

It's not merged at all with that as far as I can see. But that should be a followup.

mondrake’s picture

#52 seems overkill, since all these classes' execute methods in the end hand over the execution to Connection::query where we trigger the deprecation already.

Adding a test for \Drupal\Core\Database\Query\Select to demonstrate that.

andypost’s picture

+++ b/core/includes/database.inc
@@ -53,11 +53,9 @@
+  $target = empty($options['target']) ? 'default' : $options['target'];
+  unset($options['target']);

@@ -92,11 +90,9 @@ function db_query($query, array $args = [], array $options = []) {
+  $target = empty($options['target']) ? 'default' : $options['target'];
+  unset($options['target']);

My guts tell me to create a protected method a-la getTarget($options)

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

voleger’s picture

mondrake’s picture

Status: Needs review » Needs work
+++ b/core/lib/Drupal/Core/Database/Database.php
@@ -511,4 +511,39 @@ protected static function getDatabaseDriverNamespace(array $connection_info) {
+  /**
+   * Get target helper.
+   *
+   * Helps get "target" database from the query options.
+   *
+   * @param array $options
+   *   An array of options to control how the query operates.
+   * @param string|null $default
+   *   The default database target name.
+   * @param string[] $default_aliases
+   *   Aliases for the target that should be recognized as a default target.
+   *   For cases like target "replica" that should be changed to the "default".
+   *
+   * @return mixed|string
+   *   The target database for the database connection.
+   */
+  public static function getTarget(array &$options, $default = 'default', array $default_aliases = []) {

Honestly it looks a bit weird to add a method on the Database class that is only used in deprecated code. If we really need that, maybe a _db_get_target procedural function in database.inc could do? Then to be deprecated itself and set for removal in D9?

voleger’s picture

Status: Needs work » Needs review
FileSize
13.03 KB
9.05 KB

Addressed #57

Status: Needs review » Needs work

The last submitted patch, 58: 3000931-58.patch, failed testing. View results

voleger’s picture

db_query() still not deprecated completely, so _db_get_target() shouldn't trigger deprecation message right now.

voleger’s picture

Status: Needs work » Needs review
FileSize
12.9 KB
516 bytes

Removed trigger_error call. This will require a followup issue.

voleger’s picture

mondrake’s picture

Status: Needs review » Reviewed & tested by the community

This looks good to me. Just do not know if @alexpott's objection in #52 is properly addressed in #53. RTBCing to bring to the right eyes.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Change is hard. Nice work on a complex issue.

+++ b/core/includes/database.inc
@@ -472,10 +433,44 @@ function db_driver() {
+ * @deprecated as of Drupal 8.8.0, will be removed in Drupal 9.0.0.
+ */
+function _db_get_target(array &$options, $default = 'default', array $default_aliases = []) {

There needs to be an associated change record. Let's point at https://www.drupal.org/node/2993033.

This is a bit of a special case because we adding a deprecated function. I've debated whether this is a good idea but I think it is the best of a lot of bad options. Also I agree we shouldn't add an @trigger_error because all the methods calling it are already deprecated apart from db_query() and that's happening in another issue.

+++ b/core/includes/database.inc
@@ -472,10 +433,44 @@ function db_driver() {
+ * @param string|null $default
+ *   The default database target name.
+ * @param string[] $default_aliases
+ *   Aliases for the target that should be recognized as a default target.
+ *   For cases like target "replica" that should be changed to the "default".

I don't see why we have these two arguments and not a single boolean flag which is something like $allow_replica that defaults to TRUE and is FALSE for the couple of cases where it cannot be. The additional complexity seems unnecessary. Maybe it was when it was more generic and part of the database class but now it that does not appear to be the case.

mondrake’s picture

Assigned: Unassigned » mondrake

Working on this.

mondrake’s picture

Assigned: mondrake » Unassigned
Status: Needs work » Needs review
FileSize
5.53 KB
12.99 KB

Addressed #64

mondrake’s picture

Missed one.

mondrake’s picture

voleger’s picture

Status: Needs review » Reviewed & tested by the community

#67 addressed the #64
Looks good. +1 for rtbc

catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed 07a3820 and pushed to 8.8.x. Thanks!

  • catch committed 07a3820 on 8.8.x
    Issue #3000931 by mondrake, voleger, andypost, longwave, alexpott, git...
andypost’s picture

It needs to publish change record

voleger’s picture

Follow-up issue is unblocked and ready for the review #3049380: Complete deprecation of _db_get_target() function

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.