Problem/Motivation

Method Schema::findTables() accepts an expression as argument. Its documentation gives as example cache_% and does not mention what the expected results will be when using this string.

A developer would assume the matched tables are all prefixed with cache_ (including the underscore), while actually all tables starting with cache + one more characters are included, for example cachetags table is included while a possible cache table will not be included.

Steps to reproduce

With drush php (or in a test), in a running drupal database:

\Drupal::database()->schema()->findTables('cache_%');

Expected behaviour: all tables with cache_ - including underscore - are returned.
Actual behaviour: cachetags table is returned together with the expected behaviour results

\Drupal::database()->schema()->findTables('_%');

Expected behaviour: tables prefixed with _ are returned. In a drupal installation this should return an empty array.
Actual behaviour: all drupal tables are returned.

Proposed resolution

If this is unexpected, let's fix it. If this is expected, document the behaviour. Expected, so document it. See #8.

Remaining tasks

  • Understand if this is expected, and document it, or unexpected, and fix it
  • Update documentation, and add a test so we don't revert the behaviour by mistake.
  • User interface changes

    N/A

    API changes

    Possible changes, either in documentation or in some backwards compatibility whether the fix is disruptive for some table/db-storage scenarios.

    Data model changes

    N/A

    Release notes snippet

    If we need a Release Notes, add the snippet in here.

Comments

gambry created an issue. See original summary.

gambry’s picture

StatusFileSize
new835 bytes

Here a test-only patch.

gambry’s picture

Status: Active » Needs review

Oops. Needs Review to trigger testbot.

gambry’s picture

The test only patch in #2 should highlight the problem, although a sort() is missing. I'll upload a test-only patch with the addition after testbot job completes.

Status: Needs review » Needs work

The last submitted patch, 2: 3269091-1--test-only.patch, failed testing. View results

gambry’s picture

StatusFileSize
new854 bytes
gambry’s picture

Status: Needs work » Needs review

Setting Needs Review only to trigger testbot against patch in #6.

alexpott’s picture

Ooh... nice find. It looks like we have explicit code to support % and _

See

    // Convert the table expression from its SQL LIKE syntax to a regular
    // expression and escape the delimiter that will be used for matching.
    $table_expression = str_replace(['%', '_'], ['.*?', '.'], preg_quote($table_expression, '/'));
    $tables = preg_grep('/^' . $table_expression . '$/i', $tables);

So I think this issue should improve the documentation - which is currently:

   * @param string $table_expression
   *   An SQL expression, for example "cache_%" (without the quotes).

I think this needs to document that both _ and % are treated as wildcards and it should document their meaning.

Status: Needs review » Needs work

The last submitted patch, 6: 3269091-4--test-only.patch, failed testing. View results

gambry’s picture

Title: Unexpected/Undocumented behaviour for Schema::findTables() when an underscore is used » Undocumented behaviour for Schema::findTables() when an underscore is used
Issue summary: View changes

Updating IS to reflect this is a documentation problem.

Working on a patch.

gambry’s picture

Status: Needs work » Needs review
StatusFileSize
new3.14 KB

I went with a more-is-more approach, so document as much as possible. Also I preferred to remove 'cache_%' from the example, since it's really missleading, in favour of a more general "foo_%bar".

Does this need rollback to 9.4.x?

yogeshmpawar’s picture

StatusFileSize
new3.14 KB
new2.25 KB

Resolved CSpell errors.

Status: Needs review » Needs work

The last submitted patch, 12: 3269091-12.patch, failed testing. View results

yogeshmpawar’s picture

Status: Needs work » Needs review
StatusFileSize
new3.16 KB
new545 bytes
gambry’s picture

Status: Needs review » Needs work

Hi @yogeshmpawar . Thank you for making updates to this patch. However adding "_" to the table names triggering the spellcheck errors is not the solution. The point is exactly to test "_" behaviour as well as avoiding misunderstanding in the documentation.

What I was thinking is to use another separator other than underscores for the words triggering the errors, i.e. using numbers like "0".
I don't think foo0whatever0bar will trigger the error?

gambry’s picture

Status: Needs work » Needs review
StatusFileSize
new2.36 KB
new3.14 KB

Even better! Using camelCase is allowed, so here fix for spellcheck errors on #11 using camelcase.

jonathanshaw’s picture

All nits.

  1. +++ b/core/lib/Drupal/Core/Database/Schema.php
    @@ -180,7 +180,13 @@ public function tableExists($table) {
    +   *   An SQL expression, for example "foo_%bar" (without the quotes). Both '_'
    

    Nit: Sometimes the comment uses double quotes, sometimes single quotes

  2. +++ b/core/lib/Drupal/Core/Database/Schema.php
    @@ -180,7 +180,13 @@ public function tableExists($table) {
    +   *   any character, one occurrence) and '%' with '.*?' (hint: any character,
    

    Better "any character, exactly one occurence" or "any character, a single occurence"?

  3. +++ b/core/lib/Drupal/Core/Database/Schema.php
    @@ -180,7 +180,13 @@ public function tableExists($table) {
    +   *   any character, one occurrence) and '%' with '.*?' (hint: any character,
    +   *   multiple occurrences, lazy quantifier) before executing the regular
    

    does "any character,
    multiple occurrences, lazy quantifier" mean "any characters, zero or more occurences"?

  4. +++ b/core/lib/Drupal/Core/Database/Schema.php
    @@ -180,7 +180,13 @@ public function tableExists($table) {
    +   *   like "foo_bar", "foo__bar", "foo1bar", "fooWhateverBar", etc.
    

    It would really help if you can give an example of the difference between the % and _ matching. Also it's kind of confusing that the matches also contain the underscore.

How's this:

+   *   An SQL expression, for example 'foo%bar' (without the quotes). Both '_'
+   *   and '%' are treated as wildcards, where '_' is replaced with '.' (
+   *   any character, exactly one occurrence) and '%' with '.*?' (any character,
+   *   zero or more occurrences) before executing the regular
+   *   expression looking for matching tables names. So 'foo%bar' matches
+   *   table names like ''foobar', 'fooXbar', 'fooXBAR',  or 'fooXXXbar'; whereas 
+   *   'foo_bar' matches 'fooXbar' and 'fooXBAR' but not 'foobar' or 'fooXXXbar'.

The test looks great, this will be RTBC once the comment grammar is fixed.

joachim’s picture

A bit more nitpicking :)

  1. +++ b/core/lib/Drupal/Core/Database/Schema.php
    @@ -180,7 +180,13 @@ public function tableExists($table) {
    +   *   An SQL expression, for example "foo_%bar" (without the quotes). Both '_'
    +   *   and '%' are treated as wildcard, where '_' are replaced with '.' (hint:
    +   *   any character, one occurrence) and '%' with '.*?' (hint: any character,
    +   *   multiple occurrences, lazy quantifier) before executing the regular
    +   *   expression looking for matching tables names. So the "foo_%bar" example
    +   *   will translate to "foo..*?bar" regular expression matching table names
    +   *   like "foo_bar", "foo__bar", "foo1bar", "fooWhateverBar", etc.
    

    I'm very much in favour of 'more is more' for documentation!

    But I find the use of 'hint' here confusing.

    Also, explaining these in terms regexp syntax doesn't seem right here, when it's much simpler to say that these are wildcards used by the MySQL LIKE operator.

gambry’s picture

Status: Needs review » Needs work

Thanks both for your input.

Explaining what's happening in there can be tricky without falling in a spiral of lengthy docblock. But I like the suggestion in #17 so I'll try to apply them.

Only caveats:

does "any character, multiple occurrences, lazy quantifier" mean "any characters, zero or more occurences"?

Kind of, but we have to add the "lazy quantifier" bit at the end, which is the technical name of the "?" quantifier operator

But I find the use of 'hint' here confusing.

The only reason why I put the hint is because searching on the web for "regular expression ?" doesn't tell you what the "?" stands for. So the hints are really to avoid developers going crazy and look for what "/.*?/" syntax mean.
And since I hint for one operator I might as well hint them all.

Also, explaining these in terms regexp syntax doesn't seem right here, when it's much simpler to say that these are wildcards used by the MySQL LIKE operator.

That is what is confusing. There $table_expression argument is not used in SQL LIKE. Schema::findTables() method first load all the tables in the running database and then a regular expression extract the matching results. So we have to use regexp syntax because that is what is used to find the tables.

Needs work in order to address the feedbacks

joachim’s picture

> The only reason why I put the hint is because searching on the web for "regular expression ?" doesn't tell you what the "?" stands for.

Yup, what I meant is that writing 'hint:' reads weirdly.

> There $table_expression argument is not used in SQL LIKE. Schema::findTables() method first load all the tables in the running database and then a regular expression extract the matching results

Right, but that's an implementation detail.

What this function does is: 'You can use SQL LIKE wildcards to work with table names'. We're in the domain of SQL queries, so it makes sense that this function uses SQL wildcards. We should document like that as well.

The reason the function's code converts them to regex is that you can't actually use wildcards with table names in SQL. But this function is allowing you to pretend that you can.

jonathanshaw’s picture

https://dev.mysql.com/doc/refman/8.0/en/pattern-matching.html

SQL pattern matching enables you to use _ to match any single character and % to match an arbitrary number of characters (including zero characters). In MySQL, SQL patterns are case-insensitive by default.

Therefore I suggest:

Finds all tables that match the specified table name pattern.

@param string $table_expression
A case-insensitive pattern against which table names are compared. Both '_' and '%' are treated like wildcards in MYSQL LIKE expressions, where '_' matches any single character and '%' matches an arbitrary number of characters (including zero characters). So 'foo%bar' matches table names like ''foobar', 'fooXbar', 'fooXBAR',  or 'fooXXXbar'; whereas 'foo_bar' matches 'fooXbar' and 'fooXBAR' but not 'foobar' or 'fooXXXbar'.
joachim’s picture

#21 -- yup, perfect!

(Well, I think it's capitalised MySQL ;)

gambry’s picture

Status: Needs work » Needs review
StatusFileSize
new3.07 KB
new1.48 KB

I'm afraid there was more fun than expected. The spellcheck spit out most of those "fooXbar". I had to play with camelCase to get it working with wording from #21.

Please have another look.

jonathanshaw’s picture

Status: Needs review » Reviewed & tested by the community
+++ b/core/lib/Drupal/Core/Database/Schema.php
@@ -180,7 +180,12 @@ public function tableExists($table) {
+   *   like ''foobar', 'fooXBar', 'fooXBaR',  or 'fooXxBar'; whereas 'foo_bar'

Nit: double apostrophe

I think we're there, trusting the apostrophe can be fixed on commit.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 23: 3269091-23.patch, failed testing. View results

yogeshmpawar’s picture

Status: Needs work » Needs review
StatusFileSize
new3.07 KB
new786 bytes

Updated patch will address #24 & failed tests looks unrelated.

jonathanshaw’s picture

Status: Needs review » Reviewed & tested by the community
alexpott’s picture

Version: 10.0.x-dev » 9.3.x-dev
Status: Reviewed & tested by the community » Fixed

Committed 79925d2 and pushed to 10.0.x. Thanks!
Committed and pushed e9294d88bf to 9.4.x and 467d4fd1b3 to 9.3.x. Thanks!

  • alexpott committed 79925d2 on 10.0.x
    Issue #3269091 by gambry, yogeshmpawar, jonathanshaw, joachim, alexpott...

  • alexpott committed e9294d8 on 9.4.x
    Issue #3269091 by gambry, yogeshmpawar, jonathanshaw, joachim, alexpott...

  • alexpott committed 467d4fd on 9.3.x
    Issue #3269091 by gambry, yogeshmpawar, jonathanshaw, joachim, alexpott...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.