This issue is part of #2157455: [Meta] Make Drupal 8 work with PostgreSQL or remove support from core before release.

Currently there is one or more fail or exception in the search test group identified by the new docker based testbot that need to be fixed to make PostgreSQL finally passing all tests. Command in use:

sudo DCI_DBTYPE='pgsql' \
DCI_DBVER='9.1' \
DCI_PHPVERSION='5.5' \
DCI_CONCURRENCY='4' \
DCI_TESTGROUPS='search' \
DCI_DRUPALBRANCH='8.0.x' \
DCI_VERBOSE='true' \
DCI_UPDATEREPO='true' \
./containers/web/run.sh

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Bug because it breaks tests on PostgreSQL
Issue priority Major
Disruption Non Disruptive for core/contributed and custom modules/themes because it fixes a bug only

Comments

bzrudi71’s picture

Issue summary: View changes
StatusFileSize
new374.3 KB

GROUP BY error and some others...

bzrudi71’s picture

Status: Active » Needs review
StatusFileSize
new551 bytes

This should fix broken SearchBlockTests as a first start. Let's see what MySQL bot thinks...

bzrudi71’s picture

StatusFileSize
new374.6 KB
new375.51 KB

Great, patch fixes broken SearchBlockTest and SearchPageTextTest. However while working on the remaining issues I noticed some strange random test fails because of database is locked errors. This happens namely in SearchNumberMatchingTest and SearchRankingTest as a handful of runs show.
I also run the search test group with MySQL and the same random fails show up, so it's not really PostgreSQL only. Anyway, I have no real idea how to work around that yet, so any help is welcome here ;-)

I attached the PostgreSQL and MySQL test results for review.

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[HY000]: General
    error: 5 database is locked: INSERT INTO {simpletest} (test_id, test_class,
    status, message, message_group, function, line, file) VALUES (?, ?, ?, ?, ?,
    ?, ?, ?); Array
    (
        [0] => 22
        [1] => Drupal\search\Tests\SearchRankingTest
        [2] => pass
        [3] => Search ranking "sticky" order.
        [4] => Other
        [5] => Drupal\search\Tests\SearchRankingTest->testRankings()
        [6] => 134
        [7] => /var/www/core/modules/search/src/Tests/SearchRankingTest.php
    )
     in Drupal\simpletest\TestBase->assert() (line 292 of
    /var/www/core/modules/simpletest/src/TestBase.php).
    Drupal\Core\Database\Connection->query('INSERT INTO {simpletest} (test_id,
    test_class, status, message, message_group, function, line, file) VALUES (?,
    ?, ?, ?, ?, ?, ?, ?)', Array, Array)
    Drupal\Core\Database\Query\Insert->execute()
    Drupal\Core\Database\Driver\sqlite\Insert->execute()
    Drupal\simpletest\TestBase->assert(1, 'Search ranking "sticky" order.',
    'Other')
    Drupal\simpletest\TestBase->assertEqual('2', '2', 'Search ranking "sticky"
    order.')
    Drupal\search\Tests\SearchRankingTest->testRankings()
    Drupal\simpletest\TestBase->run()
    simpletest_script_run_one_test('22',
    'Drupal\search\Tests\SearchRankingTest')
bzrudi71’s picture

Guess this could be caused by a still running cron job? I mean, search tests make often use of cron for indexing... needs investigation

bzrudi71’s picture

Status: Needs review » Reviewed & tested by the community

I found that the 'database is locked' issue happens even outside of search test group, so that will need a separate issue. Patch still applies and passes MySQL bot so I'm setting to RTBC as it' a just single line change and does fix PG fails.

bzrudi71’s picture

bzrudi71’s picture

Priority: Normal » Major
Issue summary: View changes
webchick’s picture

Component: postgresql db driver » search.module
Status: Reviewed & tested by the community » Needs review

Please don't mark your own patch "Reviewed & tested by the community"; the purpose of that status is to signify that peer review has been completed.

The patch itself actually touches search module, so reassigning to that component. Either pwolanin or jhodgdon would be good people to reach out to for review.

bzrudi71’s picture

StatusFileSize
new383.36 KB

Okay thanks @webchick! So let's wait for a search maintainer review then... Just to make sure, attaching the PostgreSQL test run log showing 100% passing tests with patch applied.

jhodgdon’s picture

Status: Needs review » Needs work

I am having trouble understanding this.

Could you please post a shorter log that just shows the actual errors that are occurring without the patch in PostgreSQL? The full test log is huge.

I am also not sure about the patch, or whether it will work in all cases. What you've done is:

    $inner->join('search_dataset', 'd', 'i.sid = d.sid AND i.type = d.type');
     if (count($this->conditions)) {
       $inner->condition($this->conditions);
+      $inner->groupBy('i.sid');
     }

But is.sid is already in the query, even if the count($this->conditions) evaluates to FALSE. So wouldn't the group by be needed even outside the if() statement? Lower down in the same function:

    $count = db_select($inner->fields('i', array('sid')), NULL, array('target' => 'replica'));

    // Add the COUNT() expression.
    $count->addExpression('COUNT(*)');

so it seems like you would need the groupBy no matter what?

It is quite possible that our test coverage doesn't include a case where there are no conditions coming into this function... but I'm uncomfortable endorsing this fix as making the Search module actually work well with PostgreSQL in all cases. We may need more tests added to cover that bit of code.

andypost’s picture

Issue summary: View changes
StatusFileSize
new25.18 KB

Tested that,
\Drupal\search\Tests\SearchBlockTest fails when only one character passed to search, there's actually no grouping happens /search/node?keys=a

The second test is

[32mPass      Browser    SearchPageTextTes  107 Drupal\search\Tests\SearchPageTextT
[0m    Valid HTML found on "http://localhost/search/node?keys=n"
[31mFail      Other      SearchPageTextTes  108 Drupal\search\Tests\SearchPageTextT
[0m    Keyword message is displayed when searching for short word

Looks the same - one letter search

Here's debugger

jhodgdon’s picture

Hm, I'm not convinced that in that case that SearchQuery::conditions() is empty.

Can you just try moving the added group by line outside the if(), like maybe just where the COUNT is added to the query, and see if that breaks anything? I'm really still not comfortable with where this is added.

andypost’s picture

The strange thing for me is that query is run when only one char but latter the form set message that keyword should be 3 chars at least.
Maybe it's a separate bug, but it makes sense to fix that here

jhodgdon’s picture

That seems like a completely separate issue from "PostgreSQL tests fail". Let's get a separate issue for that.

bzrudi71’s picture

@jhodgdon, here is the actual error:

[0m    [03-Dec-2014 09:37:56 UTC] Uncaught PHP Exception
    Drupal\Core\Database\DatabaseExceptionWrapper: "SQLSTATE[42803]: Grouping
    error: 7 ERROR:  column "i.sid" must appear in the GROUP BY clause or be
    used in an aggregate function
[31mFail      Fatal erro Unknown              0 Unknown                            
[0m    LINE 3: (SELECT i.sid AS sid
[31mFail      Fatal erro Unknown              0 Unknown                            
[0m    ^" at /var/www/core/lib/Drupal/Core/Database/Driver/pgsql/Connection.php

Yes, adding the group by only within the if condition feels wrong, but if I remember right I already tried it outside but with no luck. I will try that again later and post the results.

bzrudi71’s picture

Category: Task » Bug report
Status: Needs work » Needs review
StatusFileSize
new720 bytes

Okay, here we go. Adding the groupBy right after! the if condition seems to work. That means we should now be save, even in case that the if condition is false...
Let's see if we pass MySQL with patch attached.

jhodgdon’s picture

Status: Needs review » Needs work

OK, great! MySQL seems to be happy too.

One nitpick:

+    // PostgreSQL requires a group by condition to prevent a GROUPING ERROR
+    $inner->groupBy('i.sid');

You need a . at the end of the comment.

Also not sure why you removed the blank line near the end of the function.

andypost’s picture

bzrudi71’s picture

Status: Needs work » Needs review
StatusFileSize
new563 bytes

New patch based on comments from #18.

jhodgdon’s picture

Status: Needs review » Reviewed & tested by the community

OK, looks good now. Thanks for the updated patch!

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

This issue addresses a major bug and is allowed per https://www.drupal.org/core/beta-changes. Committed 3ad86b6 and pushed to 8.0.x. Thanks!

@bzrudi71 is that a public postgres testbot result anywhere and if you're running the full suite against head can you publish the results somewhere. It would be great to know when we've got all of these fixed.

  • alexpott committed 3ad86b6 on 8.0.x
    Issue #2356991 by bzrudi71, andypost: PostgreSQL: Fix tests in search...
bzrudi71’s picture

@alexpott: I have just setup a public (very simple) PostgreSQL Bot on Wednesday that runs all tests once a day. BTW, thanks for commit.

andypost’s picture

jhodgdon’s picture

That patch didn't work in MySQL. But you could look at the later patch, and remove the added PostgreSQL line from this patch, and see if it still works in PostgreSQL. My guess is that it won't, because PostgreSQL does tend to be very picky about Group By.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.