Problem/Motivation

There are a lot of calls to db_query('SELECT COUNT(*) ...');.

Find for example just those in test code with find . -type f -name '*Test.php' -exec grep -Ei "QUERY.*COUNT.*\(\*\)" {} \+.

There's a more db abstract way to achieve the same with $injected_database->select(...)->countQuery()

Proposed resolution

Convert those that are not related to entity storage tables to $injected_database->select(...)->countQuery() in test code.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

mondrake created an issue. See original summary.

mondrake’s picture

Title: [PP-3] Replace all calls to db_query, which is deprecated » Convert query('SELECT COUNT(*) FROM ...') to select()
mondrake’s picture

Title: Convert query('SELECT COUNT(*) FROM ...') to select() » Convert query('SELECT COUNT(*) FROM ...') to select() in test code
mondrake’s picture

Status: Active » Needs review
StatusFileSize
new28.03 KB

First go, with DatabaseTestBase tests only.

Status: Needs review » Needs work

The last submitted patch, 4: 2994904-4.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

mondrake’s picture

Title: Convert query('SELECT COUNT(*) FROM ...') to select() in test code » Convert query('SELECT COUNT(*) FROM ...') to select('...')->countrQuery() in DatabaseTestBase tests
Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new28.19 KB
new1.25 KB

c/p error + fix to CS

andypost’s picture

It is interesting to check how count query affects performance. Probably query_alter hooks still fired if "select" query used

Some of this places using query() instead of select() to not allow alter the query with hook and save performance

mondrake’s picture

Yes performance and alterability is a topic, that's why I am proposing here to do that only in test code for the moment. BTW latest patches are doing this already, see DatabaseLegacyTest and DeleteTruncateTest.

On a general note - I think db_query and Connection::query are evil because there's always a risk that some DBMS cannot execute. Example: Oracle identifiers (table and column names, aliases) cannot be longer than 30 characters. Drupal hits this limit easily. If you hardcode identifiers in a string it's hard to work around, whereas in the select query builder you can pre and post process more easily. #2986452: Database reserved keywords need to be quoted as per the ANSI standard is suggesting to improve the situation by encapsulating column/alias identifiers in square brackets, but still.

IMHO Drupal core should 'lead the way' and put db abstraction first, and revert to SQL string statements only when performance is a hit.

mondrake’s picture

Title: Convert query('SELECT COUNT(*) FROM ...') to select('...')->countrQuery() in DatabaseTestBase tests » Convert query('SELECT COUNT(*) FROM ...') to select('...')->countQuery() in DatabaseTestBase tests
Issue summary: View changes
mondrake’s picture

Assigned: mondrake » Unassigned

Status: Needs review » Needs work

The last submitted patch, 6: 2994904-6.patch, failed testing. View results

mondrake’s picture

Issue tags: +Needs reroll, +Novice
longwave’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll, -Novice
StatusFileSize
new28.24 KB

Rerolled.

voleger’s picture

Status: Needs review » Reviewed & tested by the community

Looks good

xano’s picture

There are 44 more occurrences in test cases?

mondrake’s picture

#15, yes but let's do those in a followup, since #2991337: Document the recommended ways to obtain the database connection object is still open.

xano’s picture

Got it. Thanks for the explanation! I can confirm the remaining usages are indeed combined with db_query() and are suitable for tackling in those usages.

mondrake’s picture

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 13: 2994904-13.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Reviewed & tested by the community
StatusFileSize
new28.32 KB

Rerolled. Only diff context changes.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Let's do #3000016: Convert query('SELECT COUNT(*) FROM ...') to select('...')->countQuery() in tests (step 2) together with this splitting the issue doesn't really make sense.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new56.79 KB
new28.47 KB

Here's the merged patch.

mondrake’s picture

Title: Convert query('SELECT COUNT(*) FROM ...') to select('...')->countQuery() in DatabaseTestBase tests » Convert query('SELECT COUNT(*) FROM ...') to select('...')->countQuery() in tests
voleger’s picture

Looks like this patch cover all count query in tests.
but needs reroll

mondrake’s picture

StatusFileSize
new56.79 KB

Rerolled

voleger’s picture

Status: Needs review » Reviewed & tested by the community

+1 for RTBC

mondrake’s picture

StatusFileSize
new56.65 KB
voleger’s picture

Tests still green

berdir’s picture

Hm...

> There's a better and more db abstract way to achieve the same with $injected_database->select(...)->countQuery()

Better is actually questionable. The problem with countQuery() is that it's inefficient. It basically does SELECT COUNT(*) from ($original_query); which can get very slow with certain queries, e.g. when using group by/distinct. That implementation was done to ensure that the results are always complex, even with very complex queries. That's also why the Pager extender has a method to set a custom, optimized count query.

When to use query() vs select() has been discussed since DBTNG has been added in Drupal 7.x. Some people disagree with that, but the original intent was that select() should only be used if necessary. For example for dynamic/alterable queries or when a query can not be written in a way that works on all backends. The select query builder is quite a bit slower than a simple string obviously. Again, the least of our problems in our tests, but wanted to point that out either way..

That doesn't matter much in tests, but that might not be true at all for real queries which I think was moved to a separate issue.

Second, looking at the converted examples, I see a lot of queries against aggregator_item, aggregator_feed. Those are entity tables, and query() and select() against a content entity table are both equally wrong, if we do bother to touch those, I'd better use entity queries.

EntityApiTest is a different, as that tests the actual implementation and specific tables, it makes sense there to check the specific tables.

mondrake’s picture

Thanks @Berdir!

  1. The main purpose of this issue is to help the conversion of deprecated db_query calls. With this in, #2875394: Replace all calls to db_query, which is deprecated would be signficantly smaller.
  2. That doesn't matter much in tests, but that might not be true at all for real queries which I think was moved to a separate issue.

    Actually there is not currently any issue addressing that, and based on the comments here I think there will not be.

  3. I see a lot of queries against aggregator_item, aggregator_feed. Those are entity tables, and query() and select() against a content entity table are both equally wrong, if we do bother to touch those, I'd better use entity queries.

    Going from db_query to entity queries here seems to me too far. I'd rather leave those behind and address them in a separate issue.


  4. I basically see three options here now:
    1. Won't fix this. All db_query calls will be converted to $injected_database->query in #2875394: Replace all calls to db_query, which is deprecated
    2. Commit #20 which only converts calls in the DatabaseTestBase tests. Address the rest including conversion to entity queries in a separate issue.
    3. Commit #27, and open a follow up for the selective conversion to entity queries where appropriate.
andypost’s picture

I'd prefer to split the issue into
- convert entity related queries to entity ones, that could be a bug report
- discus the way we replace direct count queries which is fast with slow alterable "select" queries

voleger’s picture

larowlan’s picture

Status: Reviewed & tested by the community » Needs work

this is failing on Postgres?

mondrake’s picture

#34 HEAD has an intermittent failure on Postgres, there's a critical on that #2982755: Random failure in SchemaTest::testSchemaChangePrimaryKey with order of composite primary key. This patch is not relevant for that.

larowlan’s picture

Status: Needs work » Reviewed & tested by the community

thanks

catch’s picture

Status: Reviewed & tested by the community » Needs work

I agree with @andypost in #31 - where we're querying entity tables, we should just convert the whole way to an entity query. Would suggest a new issue for that, and leave the other conversions here.

mondrake’s picture

Assigned: Unassigned » mondrake
mondrake’s picture

Assigned: mondrake » Unassigned
Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -Needs issue summary update
StatusFileSize
new14.82 KB
new41.83 KB

I hope to have got it right. Updated IS. Follow-up(s) to be opened.

Remaining tests where db_query should be replaced with entity queries then are:
core/modules/aggregator/tests/src/Functional/AggregatorCronTest.php
core/modules/aggregator/tests/src/Functional/DeleteFeedTest.php
core/modules/aggregator/tests/src/Functional/ImportOpmlTest.php
core/modules/taxonomy/tests/src/Functional/TermIndexTest.php

andypost’s picture

mondrake’s picture

Patch here makes a further step and actually converts [db_]query('SELECT ...') calls to $injected_database->select(....) calls.

Excludes:

Includes:

  1. Conversion to $injected_database->update(....) for some calls to db_query('UPDATE ...') that were left out from converting earlier

Also, leveraging on the goodness of reusing SelectInterface objects, without rewriting the entire query, where appropriate (example when running a count query before and after a db operation).

mondrake’s picture

Title: Convert query('SELECT COUNT(*) FROM ...') to select('...')->countQuery() in tests » Convert query('SELECT ... FROM {xxx}') to select('xxx')->... in tests
StatusFileSize
new5.91 KB
new105.9 KB

Status: Needs review » Needs work

The last submitted patch, 42: 2994904-42.patch, failed testing. View results

andypost’s picture

Issue tags: +Needs reroll
Related issues: +#3014771: Replace queryRange call in AggregatorTestBase classes
mondrake’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
StatusFileSize
new105.96 KB

Reroll.

voleger’s picture

Status: Needs review » Reviewed & tested by the community

Looks good. +1 for RTBC

catch’s picture

Status: Reviewed & tested by the community » Needs work

I would have expected this patch to change db_query() to Connection::query()/$this->connection->query(), instead this is making static queries dynamic - we haven't deprecated static queries so not sure why it's doing that to be honest.

mondrake’s picture

Status: Needs work » Reviewed & tested by the community

@catch there's #2875394: Replace all calls to db_query, which is deprecated for that. The size of that one will be greatly reduced with this in.

This one only tackles test code. Please see #30.

If committers disagree on the general approach here, please just won't fix this issue.

catch’s picture

Status: Reviewed & tested by the community » Needs work

As you can see from the issue summary there, it's suggesting replacing db_query() with ::query(), not db_query() with ::select().

Quoting it here for quick reference:

# Replace the occurrences in code.
find ./core/lib -type f -exec sed -i -r 's/db_query\(/Database::getConnection\()->query\(/g;' {} \;
find ./core/tests -type f -not -path "./core/tests/Drupal/KernelTests/Core/Database/*" -exec sed -i -r 's/db_query\(/Database::getConnection\()->query\(/g;' {} \;
find ./core/tests/Drupal/KernelTests/Core/Database -type f -not -path "./core/tests/Drupal/KernelTests/Core/Database/SchemaTest.php" -not -path "./core/tests/Drupal/KernelTests/Core/Database/QueryTest.php" -exec sed -i -r 's/db_query\(/$this->connection->query\(/g;' {} \;
find ./core/modules -type f -exec sed -i -r 's/db_query\(/Database::getConnection\()->query\(/g;' {} \;
find ./core/tests/Drupal/KernelTests/Core/Database/LoggingTest.php -type f -exec sed -i -r "s/call_user_func_array\('db_query'/call_user_func_array\([\$this->connection, 'query']/g;" {} \;
find ./core/tests/Drupal/KernelTests/Core/Database/SchemaTest.php -type f -exec sed -i -r 's/db_query\(/Database::getConnection\()->query\(/g;' {} \;
find ./core/includes/form.inc -type f -exec sed -i -r 's/db_query\(/Database::getConnection\()->query\(/g;' {} \;
find ./core/scripts/ -type f -exec sed -i -r 's/db_query\(/\\Drupal::database\()->query\(/g;' {} \;
# Start using \Drupal::database() or inject service
find ./core/modules/tracker -type f -exec sed -i -r 's/Database::getConnection\(/\\Drupal::database\(/g;' {} \;
berdir’s picture

Status: Needs work » Closed (won't fix)

Agree that this is not necessary. Per #48, I guess we can just close this as a won't fix then and convert it in #2875394: Replace all calls to db_query, which is deprecated?