Problem/Motivation

We have support LIKE and NOT LIKE operators. But we do not have the opposite operator to REGEXP, namely, the NOT REGEXP (negated regular expression).
It can blocks others feature requests, eg. #2853188: Add negated regular expressions for views filters (string and integer). Let's add this antagonist for the complete picture of the world.

Proposed resolution

  • Add NOT REGEXP + test coverage like REGEXP.
  • Fix testRegexCondition test, because it has an incorrect comparison (sort() returns true/false instead of sorted array) and an outdated style (see #6 and #8).

Remaining tasks

Commit.

User interface changes

None. (That'll be in #2853188: Add negated regular expressions for views filters (string and integer) when we expose this to views).

API changes

None. We're just getting pgsql up to parity with mysql for this particular query operation, and fixing the tests to cover it.

We *are* converting a Kernel test into using a dataprovider, but we rename the test method so there's no confusion. It's not a base class or trait shared by tests, so this doesn't count as an API change.

Data model changes

None.

CommentFileSizeAuthor
#21 2860644-21.patch5.3 KBshashikant_chauhan
#13 interdiff.txt664 bytesAnonymous (not verified)
#13 2860644-13.patch5.3 KBAnonymous (not verified)
#8 2860644-8.patch5.29 KBdaffie
#7 interdiff.txt816 bytesAnonymous (not verified)
#7 2860644-7.patch6.09 KBAnonymous (not verified)
#6 interdiff.txt564 bytesAnonymous (not verified)
#6 2860644-6.patch6.13 KBAnonymous (not verified)
#5 2860644-5.patch6.11 KBAnonymous (not verified)
#2 2860644-2.patch4.26 KBAnonymous (not verified)
#2 2860644-2-test-only.patch3.13 KBAnonymous (not verified)
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Anonymous’s picture

vaplas created an issue. See original summary.

Anonymous’s picture

The test is based on testRegexCondition(), only the results are replaced by their opposites + used no-deprecated assert methods.

Also nit improve pattern test method testRegexCondition().

-    $query->condition('t.age', '2[6]', 'REGEXP');
-    $result = $query->execute()->fetchField();
-    $this->assertEquals($result, '26', 'Regexp with number type.');
+    $query->condition('t.age', '2[5]', 'REGEXP');
+    $result = $query->execute()->fetchCol();
+    $this->assertEquals($result, [25], 'Regexp with number type.');
amateescu’s picture

Since this patch adds a test for NOT REGEXP, we need to make sure that it works on SQLite too so I queued test runs for those environments.

daffie’s picture

Component: postgresql db driver » database system

The patch looks good to me. But I have a bit of a problem with the current testRegexCondition test and the new copied and changed a bit version testNotRegexCondition test. The test testRegexCondition is written before we changed to PHPUnit testing. For PHPUnit testing it would be best if we split testRegexCondition into two separate tests and each with a provider method. @vaplas: What is your opinion?

Anonymous’s picture

@daffie, your comment makes sense to me, definitely!

IMHO, we have 3 in 1 tests :) And only one of them is sensitive to the operator (REGEXP or NOT REGEXP). Two other relate to the processing of the regular expression as a whole, and are independent of the operator.
Such:

// Ensure that filter by "#" still works due to the quoting.
It has depend only from 'pattern' of regular expression (via SQLite reason).

// Ensure that REGEXP filter still works with no-string type field.
It has depend only from 'type of field' of regular expression (via PostgreSQL reason).

Also, during refactoring, I discovered one magical piece:

+  public function testNotRegexCondition() {
...
+    $test_groups[] = [
+      'regex' => 'Ringo|George',
+      'expected' => [
+        'George',
+        'Paul',
+      ],
+    ];

This found George. What is he doing here? Patronage by Foreach? After refactoring such a feint will not work :)

Interdiff not attached, because it does not seem helpful.

Anonymous’s picture

Patronage by Foreach?

Not. By sort(), because it return only true|false. Hence:

$this->assertEqual(count($result), count($test_group['expected']), 'Returns the expected number of rows.');
$this->assertEqual(sort($result), sort($test_group['expected']), 'Returns the expected rows.');

always pass with equal sizes.

daffie’s picture

@vaplas: Tank you for rewriting the test, but it was not in the way that I had intended. I hope you do not mind. In my way you can easily add new tests by adding them to the dataprovider. See https://phpunit.de/manual/current/en/writing-tests-for-phpunit.html#writ.... Your changes to the PostgreSQL Connection class are for me RTBC. If you like my changes to SelectTest class, you set the issue status to RTBC.

Status: Needs review » Needs work

The last submitted patch, 8: 2860644-8.patch, failed testing.

Anonymous’s picture

Status: Needs work » Needs review

Super nice, I can look at the #8 patch all day long!)

Also I was wrong when I thought that case with 'no-string' does not depend on the operator. We really need to check both operators. Sorry for the noise about this.

After #8 we lost few comments:

// Ensure that filter by "#" still works due to the quoting.
...
// Ensure that REGEXP filter still works with no-string type field.

But maybe the git logs are the best way to analyze the reason for the appearance of these cases in providerRegularExpressionCondition.

RTBC for me. But it will be great if someone will confirm this too.

Anonymous’s picture

+++ b/core/tests/Drupal/KernelTests/Core/Database/SelectTest.php
@@ -443,42 +443,40 @@ public function testRandomOrder() {
+    return[

nit typo return [.

Anonymous’s picture

I will not cry much on these comments :) I just wanted to say, that I saw it during the review and do not mind, because the git-history is better tool for this (I read about it in book "Clean Code / Robert Martin"). But I also do not mind saving comments, of course.

So, we can not just duplicate the current REGEXP test to NOT REGEXP, because this leads to a number of problems:

  • ugly copy-paste,
  • illegibility (echoes of simpletest),
  • duplication sort() error.

Hence the changes in scope.

#8 patch good fix problems with database support regular expression (complete pair REGEXP - NOT REGEXP, and correction tests). Just fix nit typo.

Anonymous’s picture

Title: Add support of NOT REGEXP operator to PostgreSQL » Add support of NOT REGEXP operator to PostgreSQL + fix testRegexCondition
Issue summary: View changes

Update IS and title, because we also fix the error in the testRegexCondition.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dww’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll

No longer applies to the 8.8.x branch.
I only care about this because it's blocking #2853188: Add negated regular expressions for views filters (string and integer), and I don't have a local pgsql environ to test with.
But maybe I'll have time for a blind reroll and see what the testbot thinks. If I get a chance to try that, I'll assign to myself.
Meanwhile, if anyone else wants to move this forward, please do. ;)

Thanks!
-Derek

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

shashikant_chauhan’s picture

dww’s picture

Status: Needs review » Reviewed & tested by the community

Fantastic, thanks @shashikant_chauhan!

I just carefully reviewed the code. The test coverage looks great (thanks @daffie and @valpas -- whom I think was the original "Anonymous" we see in this issue). For a kernel test, a data provider is fine, and doesn't incur the same costs as a data provider for a functional test.

Everything looks great in the code change itself. It's simple and clean. No code style violations, etc.

I just requeued it for a bunch of different DB configurations (most importantly, pgsql!). ;) Assuming the bot comes back all green, this is ready. If it lands in time, hopefully we can get #2853188: Add negated regular expressions for views filters (string and integer) in, too, in time for 8.9.0.

Thanks folks!
-Derek

dww’s picture

Category: Feature request » Task
Issue summary: View changes

The SQLite test failure looks like a random fail from MediaLibrary JavaScript tests. :/

Drupal\Tests\media_library\FunctionalJavascript\MediaLibraryTest
...
There was 1 failure:

1) Drupal\Tests\media_library\FunctionalJavascript\MediaLibraryTest::testWidgetUpload
Failed asserting that a NULL is not empty.

That's totally unrelated to this patch.

Meanwhile, fleshing out the summary more, and calling this a task. You could almost call it a bug that we support NOT REGEXP in mysql but not pgsql.

Still RTBC, unless the pgsql tests fail for some reason. ;)

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed 70e8937c9b to 9.0.x and 9400de336c to 8.9.x and 03bdf28929 to 8.8.x. Thanks!

  • alexpott committed 70e8937 on 9.0.x
    Issue #2860644 by daffie, shashikant_chauhan, dww: Add support of NOT...

  • alexpott committed 9400de3 on 8.9.x
    Issue #2860644 by daffie, shashikant_chauhan, dww: Add support of NOT...

  • alexpott committed 03bdf28 on 8.8.x
    Issue #2860644 by daffie, shashikant_chauhan, dww: Add support of NOT...
dww’s picture

Issue summary: View changes

Sweet, thanks! Unblocked #2853188: Add negated regular expressions for views filters (string and integer). That's pretty close to RTBC already.

Cheers,
-Derek

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.