Problem/Motivation

At the moment the class Drupal\Core\Database\Query\Condition is an generic in the sense that for all databases it is the same. In #3113403: Make Drupal\Core\Database\Query\Condition driver overridable we made the class overridable for the database driver. The by core supported driver for SQLite needs to do #2031261: Make SQLite faster by combining multiple inserts and updates in a single query. Also the database driver for MS SQL-server needs to override the generic Condition class.
Creating a new Condition object is done by running the code: new Condition('AND') or new Condition('OR'). This will result in getting only the generic Condition object and not the database driver overridable one. Therefore creating Condition object with new Condition('AND') or new Condition('OR') needs to be deprecated.

Proposed resolution

Deprecate creating an instance of the class Drupal\Core\Database\Query\Condition with the new keyword. See above for why.

Remaining tasks

TBD

User interface changes

None

API changes

Add the method Drupal\Core\Database\Query\Query::getConnection(). It will return the database connection to be used for the query. The class Drupal\Core\Database\Query\Query is the base class for: Drupal\Core\Database\Query\Delete, Drupal\Core\Database\Query\Insert, Drupal\Core\Database\Query\Merge, Drupal\Core\Database\Query\Select, Drupal\Core\Database\Query\Truncate, Drupal\Core\Database\Query\Update and Drupal\Core\Database\Query\Upsert.

Data model changes

None

Release notes snippet

Creating an instance of the class Drupal\Core\Database\Query\Condition with the new keyword is deprecated and the method Drupal\Core\Database\Query\Query::getConnection() is an API addition.

Comments

Beakerboy created an issue. See original summary.

beakerboy’s picture

beakerboy’s picture

Status: Active » Needs review
StatusFileSize
new2.09 KB

Patch for this issue.

beakerboy’s picture

Assigned: beakerboy » Unassigned
daffie’s picture

Status: Needs review » Needs work
+++ b/core/modules/views/src/Plugin/views/query/Sql.php
@@ -1111,13 +1110,26 @@ protected function buildCondition($where = 'where') {
+    $target = 'default';
+    $key = 'default';
+    // Detect an external database and set the
+    if (isset($this->view->base_database)) {
+      $key = $this->view->base_database;
+    }
+
+    // Set the replica target if the replica option is set
+    if (!empty($this->options['replica'])) {
+      $target = 'replica';
+    }
+
+    $connection = Database::getConnection($target, $key);

You have copied this part from the method query(). The problem now is that we have the same code twice. It is a bit of a no-no for programmers. See: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself. Create a new helper method and put the double code in the helper method, then use the helper method instead of the double code.

daffie’s picture

Shall we change all occurrences of new Condition in the views module in this issue. Other occurrences are:
- modules/views/src/ManyToOneHelper.php
- modules/views/src/Plugin/views/display/EntityReference.php
- modules/views/src/Plugin/views/filter/BooleanOperator.php
- modules/views/src/Plugin/views/filter/StringFilter.php

beakerboy’s picture

If I create a new helper method, does that method then need a test? I would argue that as short as this repeat is, it would not require a separate method. And since I’ve been programming for over 30 years...I have an idea of what no-nos are...no need to send me Wikipedia articles.

daffie’s picture

@beakerboy: My apologies , I was not trying to offend you.

The helper method does not need testing. Make it a public method and call it something like "getConnection()". By make it public we can use it in the plugins for the other condition changes.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

beakerboy’s picture

Status: Needs work » Needs review
StatusFileSize
new7.52 KB

Here's an updated patch with the duplicated code moved to a separate function. It looks like most of the other classes that use the Condition class already have the $connection available from dependency injection. This means the new public getConnection() function is not used outside the Sql class yet. However, it's not immediately clear to me what the best way is to fix modules/views/src/ManyToOneHelper.php. This class does not have a $this->container, so is this where the new method should be used like:

    if ($add_condition) {
      $field = $this->handler->realField;
      $connection = $this->handler->query->getConnection();
      $clause = $operator == 'or' ? $connection->condition('OR') : $connection->condition('AND');
      foreach ($this->handler->tableAliases as $value => $alias) {
        $clause->condition("$alias.$field", $value);
      }

      // implode on either AND or OR.
      $this->handler->query->addWhere($options['group'], $clause);
    }

Only the Sql class has the getConnection() function. This function is not defined in QueryPluginBase. I don't know if that would be a problem.

alexpott’s picture

  1. +++ b/core/modules/views/src/Plugin/views/filter/BooleanOperator.php
    @@ -41,6 +42,42 @@ class BooleanOperator extends FilterPluginBase {
    +  public function __construct(array $configuration, $plugin_id, $plugin_definition, Connection $connection) {
    +    parent::__construct($configuration, $plugin_id, $plugin_definition);
    +    $this->connection = $connection;
    

    We're going to need to allow this to be constructed without a Connection object for the purposes of BC. So it should be Connection $connection = NULL and then if it is NULL get the database connection using \Drupal::service().

  2. +++ b/core/modules/views/src/Plugin/views/query/Sql.php
    @@ -1095,6 +1094,22 @@ public function placeholder($base = 'views') {
     
    +  public function getConnection() {
    

    This needs documentation - and why public? I think it can be protected.

  3. Re the ManyToOneHelper - we need to change the constructor to accept a database connection and deprecate not passing one it. And then change all calls to pass one it.
beakerboy’s picture

Assigned: Unassigned » beakerboy

Thanks for the feedback. I’ll continue working on this.

andypost’s picture

StatusFileSize
new2.11 KB
new6.8 KB

Fix #11.1 and reroll for 9.1.x core/modules/views/src/Plugin/views/query/Sql.php.rej

daffie’s picture

Status: Needs review » Needs work
+++ b/core/modules/views/src/Plugin/views/filter/BooleanOperator.php
@@ -48,6 +49,26 @@ class BooleanOperator extends FilterPluginBase {
+    $instance->connection = $container->get('database');

I think this is wrong. Most of the time this will be the right database connection. It goes wrong when the view is to an external database that is of an other type then the main database. For instance the main database is MySQL and the external view is to MS SQL Server. I think the database connection to use should be: $this->view->query->getConnection(). That is also why the method should be public.

beakerboy’s picture

@daffie, the existing StringFilter already had its constructor and create() method like this. Do you think it suffers from the same issue?

lendude’s picture

+++ b/core/modules/views/src/Plugin/views/filter/BooleanOperator.php
@@ -48,6 +49,26 @@ class BooleanOperator extends FilterPluginBase {
+    $instance->connection = $container->get('database');
+    return $instance;

First off, I''m not a fan of this pattern inside the create method in core. It's great for contrib and custom code to prevent breaking on constructor changes, but core itself should not need this pattern since if it breaks any constructors inside core, we can just update them too. And we have BC patterns in place to deal with and report constructor changes. Exactly like @alexpott pointed out in #11.1

The whole 'replica' logic seems to have zero test coverage, so refactoring that and turning it into public API seems like a bad idea within the scope of this issue. So I would say keep it protected and only use it in Sql. And if we feel the need to make it public we should do it in a follow up and then add proper test coverage for this. So to answer @Beakerboy in #7 "If I create a new helper method, does that method then need a test?", I think "yes" if we want to make it public and has zero coverage currently.

The coverage really goes for all the other changes as well. Did somebody check that we have coverage for these changes? I would love to say that we have 100% coverage for all things in Views, but uhhh I'd be lying. So we should really check if we have existing coverage for all the updates, and if not, we need some tests to prove that we are not breaking anything with this (they might already exist, really don't know).

daffie’s picture

andypost’s picture

Status: Needs work » Needs review
StatusFileSize
new1.11 KB
new5.47 KB

Re-roll and a bit of clean-up

daffie’s picture

Status: Needs review » Needs work

The conditions needs to use \Drupal\views\Plugin\views\query\Sql::getConnection() as their database connection.

beakerboy’s picture

It looks like this ‘pass’ is demonstrating that a test needs to be added that uses a ‘replica‘ database.

daffie’s picture

Status: Needs work » Needs review
StatusFileSize
new12.87 KB
new8.86 KB
new13.34 KB

All the new Condition() in the views plugins are changed to using the method \Drupal\views\Plugin\views\query\Sql::getConnection(). There are tests added for the changes.

The last submitted patch, 22: 3130655-22-tests-only-should-fail.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

andypost’s picture

+++ b/core/modules/views/src/Plugin/views/query/Sql.php
@@ -1111,13 +1111,13 @@ protected function buildCondition($where = 'where') {
+    $main_group = $this->getConnection()->condition('AND');
+    $filter_group = $this->groupOperator == 'OR' ? $this->getConnection()->condition('OR') : $this->getConnection()->condition('AND');
...
     foreach ($this->$where as $group => $info) {
...
       if (!empty($info['conditions'])) {
...
+        $sub_group = $info['type'] == 'OR' ? $this->getConnection()->condition('OR') : $this->getConnection()->condition('AND');

Please revert $connection = $this->getConnection(); out of loop, no reason for extra function call

andypost’s picture

Assigned: beakerboy » Unassigned

Other then this optimization it looks awesome ready!

@daffie Thank you for great test coverage!!!

beakerboy’s picture

How should the rest of the cases similar to this be handled. I created a separate issue for the case where core’s search module uses Condition in a views handler. Should each be a separate issue, or should there be one big issue. I assume the resolution will be nearly identical to this, now that the new function was created.

daffie’s picture

StatusFileSize
new1.22 KB
new13.39 KB

For #24: Fixed. Good find @andypost! I think I was to much focused on writing the tests. :)

For #26: I think we should handle them in a separate issue. There are more problems with the those query extensions. The solution from this issue is properly be part of the solution over there, only at the moment I am not sure how exactly.

beakerboy’s picture

Status: Needs review » Reviewed & tested by the community

I’ve tested #27 on sqlserver with Drupal 9.1 and it resolves this limited issue. We need to follow up and update other views filters similarly. Entity Queries have a similar issue.

beakerboy’s picture

The HandlerBase class contains the $query object, which is defined as of type QueryPluginBase. The QueryPluginBase class does not define a getConnection() function. This is defined in the Sql class. Do we know that $this->query will always have a getConnection() function? Is there another interface that it needs to be checked against? Does getConnection() need to be added as an abstract method in QueryPluginBase?

I could be missing something.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

I think the unit testing being added here is not that great and a bit fragile. When you're mocking the world to see what class something is using without actually doing any testing of what the class is supposed to be doing then there's a problem.

I think we need to re-think how we're handling this. Given that there will plenty of code in contrib that does $conditions = new Condition('OR'); (see http://grep.xnddx.ru/search?text=new%20Condition%28&filename=&page=6) we need to deprecate instantiating the class in this way. Yes this issue is a good step on the road to doing this but if we could deprecate this then we'd not have the urge to add the unit tests and we'd be able to fix all of core in one go, not regress in core by mistake, and prompt contrib to fix themselves.

So how might we do this? I think we need to move the current \Drupal\Core\Database\Query\Condition to \Drupal\Core\Database\Query\CoreCondition and then add a new \Drupal\Core\Database\Query\Condition that extends \Drupal\Core\Database\Query\CoreCondition but issue a deprecation in the constructor.

I think this is worth it because it's is the only db class that has been used like this and we're now changing best practice.

daffie’s picture

Status: Needs work » Needs review
StatusFileSize
new57.93 KB

Created a patch for the by @alexpott suggested solution.
Removed the added tests, as @alexpott was not very happy with them.
Lets see what the testbot thinks of this patch.
I did not add an interdiff file as this a very different solution.

Status: Needs review » Needs work

The last submitted patch, 31: 3130655-31.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

andypost’s picture

123 failures looks promising but interesting

daffie’s picture

Status: Needs work » Needs review
StatusFileSize
new65.13 KB
new12.38 KB

Lets see if this patch makes the testbot more happy.

daffie’s picture

StatusFileSize
new52.01 KB
new13.65 KB

Same patch as the one from comment #34, only this one uses git "Optimize diffs for renamed and copied files" from https://www.drupal.org/documentation/git/configure. Thank you @alexpott for pointing this out.

The last submitted patch, 34: 3130655-34.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Status: Needs review » Needs work

The last submitted patch, 35: 3130655-35.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

daffie’s picture

Status: Needs work » Needs review
StatusFileSize
new18.28 KB
new56.23 KB

Added the method GetConnection() to the class Drupal\Core\Database\Query\Query. It is needed because in the EntityQuery condition class we have the Select query object and from there we need to get the connection object.
Hopefully this will make the testbot happy.

Edit: Also changed the method Drupal\Node\NodeGrantDatabaseStorage::buildGrantsQueryCondition() to a regular function instead of a static one.

mradcliffe’s picture

Status: Needs review » Needs work
  1. +++ b/core/lib/Drupal/Core/Database/Query/Condition.php
    @@ -2,418 +2,18 @@
      * Generic class for a series of conditions in a query.
      */
    -class Condition implements ConditionInterface, \Countable {
    

    I asked in Slack if we needed @deprecated annotation anymore, and it looks like there is some confusion. We do need to continue to add it per @alexpott.

  2. +++ b/core/lib/Drupal/Core/Database/Query/Condition.php
    similarity index 99%
    copy from core/lib/Drupal/Core/Database/Query/Condition.php
    
    copy from core/lib/Drupal/Core/Database/Query/Condition.php
    copy to core/lib/Drupal/Core/Database/Query/CoreCondition.php
    

    This line in the diff was easy to miss, but means that we have the same code that was "removed" in Condition. +1

daffie’s picture

Status: Needs work » Needs review
StatusFileSize
new820 bytes
new56.36 KB

For #39.1: Fixed. Added @deprecated to the docblock.

mradcliffe’s picture

Status: Needs review » Needs work

Sorry, only nitpicks and standards questions here.

  1. +++ b/core/lib/Drupal/Core/Database/Query/Condition.php
    @@ -2,418 +2,24 @@
    +   * @deprecated in drupal:9.1.0 and is removed from drupal:10.0.0. Creating an
    +   *   instance of this class is deprecated.
    

    Should this be in the class docblock instead as the entire class is deprecated? Or is only the __construct method deprecated?

    It seems there is a new phpcs rule about adding @see directly below @deprecated, which is probably the URL to the Change Record.

  2. +++ b/core/lib/Drupal/Core/Database/Query/Condition.php
    @@ -2,418 +2,24 @@
    +    @trigger_error('Creating an instance of this class is deprecated in drupal:9.1.0 and is removed in drupal:10.0.0. See https://www.drupal.org/node/3159568', E_USER_DEPRECATED);
    

    For some reason phpcs and coder (installed in vendor) isn't liking this even though it seems to be accurate. I tried to manually change it, but nothing I did made coder happy. drupalci doesn't seem to care about this from the previous test run so I don't think there's anything to change here. I think it's a bug in the sniff.

  3. +++ b/core/lib/Drupal/Core/Database/Query/Query.php
    @@ -179,4 +179,13 @@ public function &getComments() {
    +  }
     }
    

    From drupalci testbot:

    line 190	Expected 1 blank line after function; 0 found
    191	The closing brace for the class must have an empty line before it
    
daffie’s picture

Status: Needs work » Needs review
StatusFileSize
new901 bytes
new56.4 KB

@mradcliffe: Thank you for your reviews!

For #41.1 and #41.3: Fixed.

For #41.2: It is properly a bug in the sniff, only I do not run PHPCS and coder on my system.

daffie’s picture

Title: Drupal\views\Plugin\views\query\SQL using core Condition class » Deprecate creating an instance of the class Drupal\Core\Database\Query\Condition with the new keyword
Issue summary: View changes

Updating the title and the IS.

daffie’s picture

StatusFileSize
new642 bytes
new56.4 KB

Updated the added comment in the method Drupal\Core\Database\Connection::condition().

daffie’s picture

Updated the CR for the API addition for the method Drupal\Core\Database\Query\Query::getConnection().

beakerboy’s picture

Status: Needs review » Needs work

Patch fails to apply.

ravi.shankar’s picture

Status: Needs work » Needs review
StatusFileSize
new69.52 KB

Added reroll of patch #44.

daffie’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll

@ravi.shankar: Thank you for rerolling the patch. Only when you look at the size differences between my patch and your patch, your patch is 13KB bigger. To fix it you must create the patch the following change in your .gitconfig file:

[diff]
  renames = copies

For more info see: https://www.drupal.org/documentation/git/configure. And yes I made the same mistake myself. :)

ravi.shankar’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
StatusFileSize
new56.4 KB

@daffie Thank you so much.

I have again added a reroll of patch #44.

beakerboy’s picture

Status: Needs review » Reviewed & tested by the community

I've tested the patch on sql server and everything runs as expected. Code looks good upon review.

daffie’s picture

@beakerboy: Thank you for the review and the RTBC!

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/core/lib/Drupal/Core/Database/Connection.php
    @@ -1164,6 +1165,12 @@ public function schema() {
    +    // Creating an instance of the class Drupal\Core\Database\Query\Condition
    +    // with the new keyword is deprecated. Instead use the class
    +    // Drupal\Core\Database\Query\CoreCondition.
    

    This comment is a bit misleading. It should be more about the why and less of the how.

  2. +++ b/core/lib/Drupal/Core/Database/Query/Condition.php
    @@ -2,418 +2,26 @@
    +class Condition extends CoreCondition {
    

    Doing some thinking. It'd be great to avoid adding the CoreCondition class. And I think we can. We can add an extra argument to the constructor of $trigger_deprecation and default it to TRUE. In \Drupal\Core\Database\Connection::condition() we can pass false to this and not trigger the deprecation.

    That way we've not added a class which we have to think about from an API perspective.

  3. +++ b/core/lib/Drupal/Core/Database/Query/Condition.php
    @@ -2,418 +2,26 @@
    +    @trigger_error('Creating an instance of this class is deprecated in drupal:9.1.0 and is removed in drupal:10.0.0. See https://www.drupal.org/node/3159568', E_USER_DEPRECATED);
    

    This message should tell people what to do. Ie.

    'Creating an instance of this class is deprecated in drupal:9.1.0 and is removed in drupal:10.0.0. Use Database::getConnection()->condition() instead. See https://www.drupal.org/node/3159568'

daffie’s picture

Status: Needs work » Needs review
StatusFileSize
new21.68 KB
new39.65 KB

Fixed all 3 points from @alexpott in comment #53.

beakerboy’s picture

Status: Needs review » Reviewed & tested by the community

I like this approach better. There’s no new class floating around, and someone can’t just change “new Condition” to “new CoreCondition”. The patch successfully runs on sqlsrv.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed f124d52 and pushed to 9.1.x. Thanks!

  • alexpott committed f124d52 on 9.1.x
    Issue #3130655 by daffie, andypost, Beakerboy, ravi.shankar, alexpott,...
joseph.olstad’s picture

Great work on this, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.