Closed (won't fix)
Project:
Drupal core
Version:
8.7.x-dev
Component:
database system
Priority:
Normal
Category:
Task
Assigned:
Unassigned
Issue tags:
Reporter:
Created:
24 Aug 2018 at 14:29 UTC
Updated:
24 Feb 2019 at 14:48 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #2
mondrakeComment #3
mondrakeComment #4
mondrakeFirst go, with
DatabaseTestBasetests only.Comment #6
mondrakec/p error + fix to CS
Comment #7
andypostIt is interesting to check how count query affects performance. Probably query_alter hooks still fired if "select" query used
Some of this places using query() instead of select() to not allow alter the query with hook and save performance
Comment #8
mondrakeYes performance and alterability is a topic, that's why I am proposing here to do that only in test code for the moment. BTW latest patches are doing this already, see DatabaseLegacyTest and DeleteTruncateTest.
On a general note - I think db_query and Connection::query are evil because there's always a risk that some DBMS cannot execute. Example: Oracle identifiers (table and column names, aliases) cannot be longer than 30 characters. Drupal hits this limit easily. If you hardcode identifiers in a string it's hard to work around, whereas in the select query builder you can pre and post process more easily. #2986452: Database reserved keywords need to be quoted as per the ANSI standard is suggesting to improve the situation by encapsulating column/alias identifiers in square brackets, but still.
IMHO Drupal core should 'lead the way' and put db abstraction first, and revert to SQL string statements only when performance is a hit.
Comment #9
mondrakeComment #10
mondrakeComment #12
mondrakeComment #13
longwaveRerolled.
Comment #14
volegerLooks good
Comment #15
xanoThere are 44 more occurrences in test cases?
Comment #16
mondrake#15, yes but let's do those in a followup, since #2991337: Document the recommended ways to obtain the database connection object is still open.
Comment #17
xanoGot it. Thanks for the explanation! I can confirm the remaining usages are indeed combined with
db_query()and are suitable for tackling in those usages.Comment #18
mondrake@Xano opened #3000016: Convert query('SELECT COUNT(*) FROM ...') to select('...')->countQuery() in tests (step 2) for the rest conversions.
Comment #20
mondrakeRerolled. Only diff context changes.
Comment #21
alexpottLet's do #3000016: Convert query('SELECT COUNT(*) FROM ...') to select('...')->countQuery() in tests (step 2) together with this splitting the issue doesn't really make sense.
Comment #22
mondrakeHere's the merged patch.
Comment #23
mondrakeComment #24
volegerLooks like this patch cover all count query in tests.
but needs reroll
Comment #25
mondrakeRerolled
Comment #26
voleger+1 for RTBC
Comment #27
mondrakeReroll after #2873684: Replace all calls to db_select, which is deprecated. Only automerge.
Comment #28
volegerTests still green
Comment #29
berdirHm...
> There's a better and more db abstract way to achieve the same with $injected_database->select(...)->countQuery()
Better is actually questionable. The problem with countQuery() is that it's inefficient. It basically does SELECT COUNT(*) from ($original_query); which can get very slow with certain queries, e.g. when using group by/distinct. That implementation was done to ensure that the results are always complex, even with very complex queries. That's also why the Pager extender has a method to set a custom, optimized count query.
When to use query() vs select() has been discussed since DBTNG has been added in Drupal 7.x. Some people disagree with that, but the original intent was that select() should only be used if necessary. For example for dynamic/alterable queries or when a query can not be written in a way that works on all backends. The select query builder is quite a bit slower than a simple string obviously. Again, the least of our problems in our tests, but wanted to point that out either way..
That doesn't matter much in tests, but that might not be true at all for real queries which I think was moved to a separate issue.
Second, looking at the converted examples, I see a lot of queries against aggregator_item, aggregator_feed. Those are entity tables, and query() and select() against a content entity table are both equally wrong, if we do bother to touch those, I'd better use entity queries.
EntityApiTest is a different, as that tests the actual implementation and specific tables, it makes sense there to check the specific tables.
Comment #30
mondrakeThanks @Berdir!
db_querycalls. With this in, #2875394: Replace all calls to db_query, which is deprecated would be signficantly smaller.Actually there is not currently any issue addressing that, and based on the comments here I think there will not be.
Going from
db_queryto entity queries here seems to me too far. I'd rather leave those behind and address them in a separate issue.db_querycalls will be converted to$injected_database->queryin #2875394: Replace all calls to db_query, which is deprecatedComment #31
andypostI'd prefer to split the issue into
- convert entity related queries to entity ones, that could be a bug report
- discus the way we replace direct count queries which is fast with slow alterable "select" queries
Comment #32
volegerComment #33
volegerComment #34
larowlanthis is failing on Postgres?
Comment #35
mondrake#34 HEAD has an intermittent failure on Postgres, there's a critical on that #2982755: Random failure in SchemaTest::testSchemaChangePrimaryKey with order of composite primary key. This patch is not relevant for that.
Comment #36
larowlanthanks
Comment #37
catchI agree with @andypost in #31 - where we're querying entity tables, we should just convert the whole way to an entity query. Would suggest a new issue for that, and leave the other conversions here.
Comment #38
mondrakeComment #39
mondrakeI hope to have got it right. Updated IS. Follow-up(s) to be opened.
Remaining tests where db_query should be replaced with entity queries then are:
core/modules/aggregator/tests/src/Functional/AggregatorCronTest.php
core/modules/aggregator/tests/src/Functional/DeleteFeedTest.php
core/modules/aggregator/tests/src/Functional/ImportOpmlTest.php
core/modules/taxonomy/tests/src/Functional/TermIndexTest.php
Comment #40
andypostFiled follow-up #3012599: Replace all db calls to aggregator_feed and aggregator_item tables with Entity APIs
Comment #41
mondrakePatch here makes a further step and actually converts
[db_]query('SELECT ...')calls to$injected_database->select(....)calls.Excludes:
$injected_database->query(....)itselfIncludes:
$injected_database->update(....)for some calls todb_query('UPDATE ...')that were left out from converting earlierAlso, leveraging on the goodness of reusing SelectInterface objects, without rewriting the entire query, where appropriate (example when running a count query before and after a db operation).
Comment #42
mondrakeAdding one more conversion from #3012599: Replace all db calls to aggregator_feed and aggregator_item tables with Entity APIs where it did not fit.
Comment #44
andypostComment #45
mondrakeReroll.
Comment #46
volegerLooks good. +1 for RTBC
Comment #47
catchI would have expected this patch to change db_query() to Connection::query()/$this->connection->query(), instead this is making static queries dynamic - we haven't deprecated static queries so not sure why it's doing that to be honest.
Comment #48
mondrake@catch there's #2875394: Replace all calls to db_query, which is deprecated for that. The size of that one will be greatly reduced with this in.
This one only tackles test code. Please see #30.
If committers disagree on the general approach here, please just won't fix this issue.
Comment #49
catchAs you can see from the issue summary there, it's suggesting replacing db_query() with ::query(), not db_query() with ::select().
Quoting it here for quick reference:
Comment #50
berdirAgree that this is not necessary. Per #48, I guess we can just close this as a won't fix then and convert it in #2875394: Replace all calls to db_query, which is deprecated?