Problem/Motivation

We should be using static queries as little as possible, because they can result into problems for a some databases. A static query may work on a number of databases, but not on an other one. Dynamic queries are a little bit slower then static queries and for use in tests that is not a problem.

Proposed resolution

Change the static queries in the directory "core/tests/Drupal/FunctionalTests" and its sub-directories. Also do the same when in the directory "core/tests/Drupal/KernelTests" and its sub-directories, but not in the directory "core/tests/Drupal/KernelTests/Core/Database".

What needs to be changed:

From:
$connection->query('SELECT nid FROM {node} WHERE nid = :id', [':id' => 1']);

To:
$connection->select('node')
  ->fields('node', ['nid'])
  ->condition('nid', 1)
  ->execute();

Remaining tasks

TBD

User interface changes

None

API changes

None

Data model changes

None

Release notes snippet

TBD

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

daffie created an issue. See original summary.

rik-dev’s picture

Assigned: Unassigned » rik-dev
mondrake’s picture

Personally, absolutely +1 on the motivation here, but please anyone working on this be aware this was attempted in #2994904: Convert query('SELECT ... FROM {xxx}') to select('xxx')->... in tests, and finally rejected. Wish you better luck :)

rik-dev’s picture

Assigned: rik-dev » Unassigned
adityasingh’s picture

Assigned: Unassigned » adityasingh
daffie’s picture

Category: Task » Bug report
Issue tags: +Bug Smash Initiative

The parent issue is a "bug report", therefore this issue is one too.

adityasingh’s picture

Assigned: adityasingh » Unassigned

Unassigning from myself because i am stuck in other issues.

siddhant.bhosale’s picture

Assigned: Unassigned » siddhant.bhosale
siddhant.bhosale’s picture

Assigned: siddhant.bhosale » Unassigned
Status: Active » Needs review
FileSize
10.38 KB

I have replaced the static queries with dynamic queries from core/tests/Drupal/FunctionalTests and its sub-folder, core/tests/Drupal/KernelTests and its sub-folders. Please review.

adityasingh’s picture

Hi @siddhant.bhosale thanks for patch
Patch looks good for me and cleanly applied.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10632  100 10632    0     0  20546      0 --:--:-- --:--:-- --:--:-- 20525
Checking patch core/tests/Drupal/KernelTests/Core/Command/DbDumpTest.php...
Checking patch core/tests/Drupal/KernelTests/Core/Entity/EntityApiTest.php...
Checking patch core/tests/Drupal/KernelTests/Core/Entity/RevisionableContentEntityBaseTest.php...
Checking patch core/tests/Drupal/KernelTests/Core/KeyValueStore/GarbageCollectionTest.php...
Checking patch core/tests/Drupal/KernelTests/Core/Routing/MatcherDumperTest.php...
Checking patch core/tests/Drupal/KernelTests/KernelTestBaseTest.php...
Applied patch core/tests/Drupal/KernelTests/Core/Command/DbDumpTest.php cleanly.
Applied patch core/tests/Drupal/KernelTests/Core/Entity/EntityApiTest.php cleanly.
Applied patch core/tests/Drupal/KernelTests/Core/Entity/RevisionableContentEntityBaseTest.php cleanly.
Applied patch core/tests/Drupal/KernelTests/Core/KeyValueStore/GarbageCollectionTest.php cleanly.
Applied patch core/tests/Drupal/KernelTests/Core/Routing/MatcherDumperTest.php cleanly.
Applied patch core/tests/Drupal/KernelTests/KernelTestBaseTest.php cleanly.
siddhant.bhosale’s picture

@adityasingh Thanks for reviewing the patch. Can you please change the status to RTBC ?

daffie’s picture

Status: Needs review » Needs work

Patch look good, only I have some remarks:

  1. +++ b/core/tests/Drupal/KernelTests/Core/Command/DbDumpTest.php
    @@ -231,7 +231,7 @@ public function testScriptLoad() {
    +    $config = unserialize($connection->select('config', 'c')->fields('c', ['data'])->condition('c.name', 'test_config')->execute()->fetchField());
    

    You do not need to say 'c.name'. Just 'name' is enough.

  2. +++ b/core/tests/Drupal/KernelTests/Core/KeyValueStore/GarbageCollectionTest.php
    @@ -21,6 +21,9 @@ class GarbageCollectionTest extends KernelTestBase {
    +  /**
    +   *
    +   */
       protected function setUp(): void {
    

    This change is out of scope and wrong. It is an empty docblock.

  3. +++ b/core/tests/Drupal/KernelTests/Core/KeyValueStore/GarbageCollectionTest.php
    @@ -60,11 +63,7 @@ public function testGarbageCollection() {
    +    $result = $connection->select('key_value_expire', 'kvp')->fields('kvp', ['name'])->condition('kvp.collection', $collection)->execute()->fetchAll();
    

    Can we make change a multiline change for better readability. Also 'kvp.collection' can be changed to just 'collection'.

  4. +++ b/core/tests/Drupal/KernelTests/KernelTestBaseTest.php
    @@ -297,11 +297,11 @@ protected function tearDown(): void {
    +        ->condition('sql_m.type', 'table')
    +        ->condition('sql_m.name', '%', 'LIKE')
    +        ->condition('sql_m.name', 'sqlite_%', 'NOT LIKE')
    

    The part "sql_m." an be removed from each line.

  5. I am still finding static queries in the following files:
    - core/tests/Drupal/FunctionalTests/Installer/InstallerDatabaseErrorMessagesTest.php;
    - core/tests/Drupal/FunctionalTests/Installer/InstallerTranslationTest.php;
    - core/tests/Drupal/KernelTests/Core/Config/Storage/DatabaseStorageTest.php;
    - core/tests/Drupal/FunctionalTests/Update/UpdatePathTestBaseTest.php.
ravi.shankar’s picture

Assigned: Unassigned » ravi.shankar

Working on this.

ravi.shankar’s picture

Assigned: ravi.shankar » Unassigned
FileSize
13.02 KB
5.65 KB

Here I have tried to address comment #12.

It still needs work to remove the static query from:
-core/tests/Drupal/FunctionalTests/Update/UpdatePathTestBaseTest.php

rajandro’s picture

Assigned: Unassigned » rajandro

I am working on this

daffie’s picture

Patch looks good:

  1. +++ b/core/tests/Drupal/KernelTests/Core/KeyValueStore/GarbageCollectionTest.php
    @@ -60,11 +60,10 @@ public function testGarbageCollection() {
    +      ->fields('kvp', ['name'])->condition('collection', $collection)
    

    Can the ->fields() part and the ->condition() part have their own line.

  2. +++ b/core/tests/Drupal/KernelTests/Core/Routing/MatcherDumperTest.php
    @@ -126,7 +129,7 @@ public function testDump() {
    +    $record = $connection->select('test_routes', 'tr')->fields('tr')->condition('tr.name', 'test_route')->execute()->fetchObject();
    

    Can we change 'tr.name' to just 'name' in the condition method.

  3. In the test core/tests/Drupal/FunctionalTests/Installer/InstallerTranslationTest.php there is the query "->query('DROP TABLE {drupal_install_test}')", that needs to be changed.
  4. In the test core/tests/Drupal/FunctionalTests/Update/UpdatePathTestBaseTest.php there is the query "->query("SELECT name, route from {router}")", that needs to be changed.
Hardik_Patel_12’s picture

FileSize
14.04 KB
1.92 KB

Kindly review a patch , covered points #16.1 , #16.2 and #16.4 as suggested by @daffie. #16.3 is still needs work.

Hardik_Patel_12’s picture

FileSize
2.25 KB

Kindly find proper interdiff , last one was diff of diff kindly avoid that.

daffie’s picture

+++ b/core/tests/Drupal/KernelTests/Core/Routing/MatcherDumperTest.php
@@ -129,7 +129,11 @@
+      ->condition('tr.name', 'test_route')

Can the 'tr.name' be changed to just 'name'.

Hardik_Patel_12’s picture

FileSize
14.04 KB
514 bytes

I don't know somehow i have missed this , changing now 'tr.name' to just 'name'.

rajandro’s picture

Hi @Hardik_Patel_12, I was working on the same issue as you can notice on #15! However, I have checked your patch as well and this is looking fine to me. I am adding the missing point #16.3 and moving this it NR.

daffie’s picture

Status: Needs review » Reviewed & tested by the community

All my points are addressed.
All code changes look good to me.
For me it is RTBC.

Hardik_Patel_12’s picture

@rajandro , this was my bad but it was unintentional. Will make sure in future that if someone is working on some issue then not to pick that issue.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 21: 3152398-21.patch, failed testing. View results

daffie’s picture

Status: Needs work » Reviewed & tested by the community

Back to RTBC.

alexpott credited Berdir.

alexpott credited andypost.

alexpott credited catch.

alexpott credited longwave.

alexpott credited voleger.

alexpott’s picture

Title: Change static queries to dynamic queries in core/tests/Drupal » [backport] Change static queries to dynamic queries in core/tests/Drupal
Version: 9.1.x-dev » 9.0.x-dev

Committed 65656ac660 and pushed to 9.1.x. Thanks!

I'll backport to 9.0.x and maybe 8.9.x if the tests are green to keep tests aligned.

Re #3 I think the statement in the issue summary that using query strings like this makes it harder for contrib db drivers it true so yeah let's do this now and credit people who worked on #2994904: Convert query('SELECT ... FROM {xxx}') to select('xxx')->... in tests

+++ b/core/tests/Drupal/KernelTests/Core/KeyValueStore/GarbageCollectionTest.php
@@ -46,12 +46,12 @@ public function testGarbageCollection() {
         ->keys([
-            'name' => 'key_' . $i,
-            'collection' => $collection,
-          ])
+          'name' => 'key_' . $i,
+          'collection' => $collection,
+        ])
         ->fields([
-            'expire' => REQUEST_TIME - 1,
-          ])
+          'expire' => REQUEST_TIME - 1,
+        ])

+++ b/core/tests/Drupal/KernelTests/Core/Routing/MatcherDumperTest.php
@@ -33,6 +33,9 @@ class MatcherDumperTest extends KernelTestBase {
+  /**
+   * {@inheritdoc}
+   */

+++ b/core/tests/Drupal/KernelTests/KernelTestBaseTest.php
@@ -224,7 +224,7 @@ public function testBootKernel() {
-    // The 'Australia/Sydney' time zone is set in core/tests/bootstrap.php
+    // The 'Australia/Sydney' time zone is set in core/tests/bootstrap.php.

@@ -323,7 +323,7 @@ public function testProfileModules() {
    *
    * @group legacy
    * @expectedDeprecation AssertLegacyTrait::assertIdenticalObject() is deprecated in drupal:8.0.0 and is removed from drupal:10.0.0. Use $this->assertEquals() instead. See https://www.drupal.org/node/3129738
-    */
+   */

Out-of-scope changes. While these changes are correct they are out-of-scope for changing static queries to dynamic queries. I've reverted them from the committed patch. Please don't make such changes in future patches it increases the amount a reviewer has to review.

  • alexpott committed 65656ac on 9.1.x
    Issue #3152398 by Hardik_Patel_12, ravi.shankar, rajandro, siddhant....

  • alexpott committed a178f53 on 9.1.x
    Revert "Issue #3152398 by Hardik_Patel_12, ravi.shankar, rajandro,...
alexpott’s picture

Title: [backport] Change static queries to dynamic queries in core/tests/Drupal » Change static queries to dynamic queries in core/tests/Drupal
Status: Reviewed & tested by the community » Needs work

I had to revert this issue because it broke sqlite testing...

14) Drupal\KernelTests\KernelTestBaseTest::testAssertIdenticalObject
Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[HY000]: General
error: 1 no such table: test75591109.test75591109sqlite_master: SELECT
"sql_m"."name" AS "name"
FROM
{test75591109sqlite_master} "sql_m"
WHERE ("type" = :db_condition_placeholder_0) AND ("name" LIKE
:db_condition_placeholder_1 ESCAPE '\') AND ("name" NOT LIKE
:db_condition_placeholder_2 ESCAPE '\'); Array
(
    [:db_condition_placeholder_0] => table
    [:db_condition_placeholder_1] => %
    [:db_condition_placeholder_2] => sqlite_%
)


/Users/alex/dev/sites/drupal8alt.dev/core/lib/Drupal/Core/Database/Connection.php:857
/Users/alex/dev/sites/drupal8alt.dev/core/lib/Drupal/Core/Database/Driver/sqlite/Connection.php:359
/Users/alex/dev/sites/drupal8alt.dev/core/lib/Drupal/Core/Database/Connection.php:821
/Users/alex/dev/sites/drupal8alt.dev/core/lib/Drupal/Core/Database/Query/Select.php:510
/Users/alex/dev/sites/drupal8alt.dev/core/tests/Drupal/KernelTests/KernelTestBaseTest.php:304
/Users/alex/dev/sites/drupal8alt.dev/vendor/phpunit/phpunit/src/Framework/TestResult.php:691
alexpott’s picture

Version: 9.0.x-dev » 9.1.x-dev
alexpott’s picture

alexpott’s picture

This should make the following change to \Drupal\KernelTests\KernelTestBaseTest::tearDown

    if ($connection->databaseType() === 'sqlite') {
      $result = $connection->query("SELECT name FROM " . $this->databasePrefix . ".sqlite_master WHERE type = :type AND name LIKE :table_name AND name NOT LIKE :pattern", [
        ':type' => 'table',
        ':table_name' => '%',
        ':pattern' => 'sqlite_%',
      ])->fetchAllKeyed(0, 0);

      $this->assertTrue(empty($result), 'All test tables have been removed.');
    }
    else {
      $tables = $connection->schema()->findTables($this->databasePrefix . '%');
      $this->assertTrue(empty($tables), 'All test tables have been removed.');
    }

i.e. swap the if around. We can use a static query here because we're hardcoding the db type. And the if this way around makes it more obvious.

daffie’s picture

This is something a novice can do. Make the change to the method tearDown() in the file "core/tests/Drupal/KernelTests/KernelTestBaseTest.php" as suggested by @alexpott. The query does not need to be changed to a dynamic one, because it is only run for SQLite.

pratik_kamble’s picture

Issue tags: +DIACWJuly2020
pratik_kamble’s picture

andypost’s picture

Would be great to elaborate in IS why change to alterable query makes things less fragile.
In my experience dynamic query (especially to entity tables) are evil because some unpredictable alters are fire and every query needs to skip access when running tests.

ultrabob’s picture

I can't address #41, but I'll work on #37

daffie’s picture

Would be great to elaborate in IS why change to alterable query makes things less fragile.

I am not sure what you would like @andypost. We are changing the queries to dynamic so that they work for contrib database drivers. Some static queries do not work for all contrib database drivers. With dynamic queries that is not a problem.

In my experience dynamic query (especially to entity tables) are evil because some unpredictable alters are fire and every query needs to skip access when running tests.

Queries on entity tables should as much as possible be changed to EntityQueries. We have an issue for that. See #3014948: [META] Replace all direct db calls to entity tables with Entity API.

ultrabob’s picture

I've made the changes suggested by @alexpott in #37.

daffie’s picture

Status: Needs review » Reviewed & tested by the community

The change looks good to me.
It is the change that was requested by @alexpott.
Back to RTBC.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed fe38a9c and pushed to 9.1.x. Thanks!

  • alexpott committed fe38a9c on 9.1.x
    Issue #3152398 by Hardik_Patel_12, alexpott, rajandro, ravi.shankar,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.