Meta issue: fix the remaining PostgreSQL bugs

The patch for Field attach tests (storage-related) is at #706248: field_sql_storage_field_storage_query() with 'count' option breaks on PostgreSQL and SQLite , awaiting review. I've also added a handbook page at http://drupal.org/node/706264 describing the problem here which has also occured in other code which blows up on Postgres.

A lot of the exceptions in Fields are due to an attempt to create a table which already exists. I'm not sure whether this is because it isn't being dropped when it should or whether there should be a check before creation.

Not critical. Please read and understand Priority levels of Issues, thanks.

Please understand the implications of the test failures.

These are not testing failures, the underlying functions (at least for the ones I've looked at) are also failing. This means that, for instance, fields cannot be added to node types, comments fail to display, and some node save operations fail. Which is the very definition of critical.

Postgres is a supported database, the fact that these functions work on more permissive MySQL doesn't make their failure on Postgres any less critical.

Agreed on the critical level.

Here is a full test log from this morning. There is a few fatal errors that needs to be investigated.

Here are the remaining test failures this morning.

Test summary (only show failed tests)
-------------

Comment paging settings 203 passes, 3 fails, and 0 exceptions
Schema API 21 passes, 5 fails, and 0 exceptions
Field attach tests (storage-related) 91 passes, 0 fails, and 1 exception
Field CRUD tests 47 passes, 0 fails, and 1 exception
List field 11 passes, 0 fails, and 1 exception
File field validation tests 91 passes, 0 fails, and 1 exception
Session https handling 52 passes, 1 fail, and 1 exception

Test run duration: 1 hour 29 min

@Josh: I'm running PostgreSQL data directory on a tmpfs mount. It helps a lot.

The comment failures are due to #314349: PDO Exception when having too much comments.

The schema failures are due to #715906: Schema tests fail on PostgreSQL.

The session HTTPS handling failures are due to #715108: Make Merge queries more consistent and robust.

All the others are field failures!

By the way, I'm seeing exactly the same field failures on SQLite:

Field attach tests (storage-related) 91 passes, 0 fails, and 1 exception
Field CRUD tests 46 passes, 1 fail, and 1 exception
List field 11 passes, 0 fails, and 1 exception
Field UI tests 102 passes, 2 fails, and 1 exception
File field validation tests 91 passes, 0 fails, and 1 exception

(with the addition of the Field UI tests that pass on PostgreSQL)

As of this morning:

Test summary (showing only failed tests)
-------------

Field CRUD tests 47 passes, 0 fails, and 1 exception
List field 11 passes, 0 fails, and 1 exception
File field validation tests 91 passes, 0 fails, and 1 exception
Session https handling 52 passes, 1 fail, and 1 exception

Test run duration: 1 hour 28 min

We are getting there, slowly.

List field and File field validation tests caused by #718016: DatabaseSchema_pgsql::renameTable() needs to rename the indexes

EDIT: all first three cases caused by exception when field_create_field() called for field which was deleted

SQLSTATE[42P07]: Duplicate table: 7 ERROR: relation "simpletest727934field_data_field_test_etid_idx" already exists

'File field validation tests' is broken because table-name is truncated to 64 chars which causes duplicate

SQLSTATE[42P07]: Duplicate table: 7 ERROR: relation "simpletest322535field_data_field_s322535u2crryva_field_s322535u" already exists

$field_name = 'field_' . strtolower($this->randomName());

produces tablename longer then 64 chars so it truncated

I file an issue #718254: Fieldname in tests are too long to change fieldname used in test

With #718016: DatabaseSchema_pgsql::renameTable() needs to rename the indexes applied:

Test summary (showing only failed tests)
-------------

File field validation tests 91 passes, 0 fails, and 1 exception
Session https handling 52 passes, 1 fail, and 1 exception

Test run duration: 1 hour 29 min

This is not critical anymore.

another bug #720620: indexExists() for pgsql and sqlite does not prefix tablename

finally commited #718254: Fieldname in tests are too long

so only one test is broken - sessions #715108: Make Merge queries more consistent and robust

grouping #727548: administration screen broken under pgsql - non-aggregated fields must be present in the GROUP BY clause

Also comments tests (anonymous comment, commrnt preview) throws exception but I cant reproduce on live site...

SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input syntax for integer: "1#comment-1": SELECT revision.timestamp AS revision_timestamp, revision.uid AS revision_uid, revision.vid AS vid, revision.title AS title, revision.log AS log, revision.status AS status, revision.comment AS comment, revision.promote AS promote, revision.sticky AS sticky, base.nid AS nid, base.type AS type, base.language AS language, base.uid AS uid, base.created AS created, base.changed AS changed, base.tnid AS tnid, base.translate AS translate FROM {node} base INNER JOIN {node_revision} revision ON revision.vid = base.vid WHERE (base.nid IN (:db_condition_placeholder_0)) ; Array ( [:db_condition_placeholder_0] => 1#comment-1 )

POST http://d7pg/comment/reply/1 returned 500 (2.89 KB).	Browser	comment.test	63	CommentHelperCase->postComment()

Issue to solve #671520: Curl and fragment URLs break testing

I think my issue in Views is related this this one:

• Error: PDOException: SQLSTATE[22P02] & Exception: SQLSTATE[42703]

Another one waiting for final review #761976: DatabaseSchema_pgsql::changeField() is broken

With Drupal 7 RC1 approaching, what is the status of this issue. It appears that the issues listed here are closed, but this probably needs one last look and then closed if everything is complete.

I've started a test suite run with the patch in #30 of http://drupal.org/node/926636

Environment:
Ubuntu 10.10
PostgreSQL 9.0.1
PHP 5.3.3
Apache 2.2.16

There are 11 tests currently not working on PostgreSQL:

Basic Upgrade Path
Comment Upgrade Path
Filter administration functionality
Filter format upgrade path
Element processing
Locale upgrade path
Node body upgrade path
Poll upgrade path
RDF comment mapping
Taxonomy upgrade path
Upload upgrade path

With the following patches applied the following tests remain broken:
#926636: Drupal install error on PostgreSQL 9.0 database
#974250: Upgrade path broken on PostgreSQL: can't change NOT NULL

- Filter administration functionality
- Filter format upgrade path
- Element Processing
- Node body upgrade path
- RDF comment mapping
- Taxonomy upgrade path

Most of the upgrade path issues are a consequence of #951116: db_change_field() fails to convert int to varchar on PostgreSQL

Allright, so all of the upgrade path issues are a consequence of last mentioned issue.

This lists the issues with the tests they fix:
#926636: Drupal install error on PostgreSQL 9.0 database
#974250: Upgrade path broken on PostgreSQL: can't change NOT NULL
Basic Upgrade Path
Comment Upgrade Path
Locale upgrade path
Poll upgrade path
Upload upgrade path

#951116: db_change_field() fails to convert int to varchar on PostgreSQL
Filter format upgrade path
Node body upgrade path
Taxonomy upgrade path

#977596: Text format loses weight and status when saving settings
Filter administration functionality

#977184: FormElementTestCase::testOptions() should fail, doesn't
Element processing

#933856: xpath() return values are inconsistent
RDF comment mapping

Update: A full test suite run confirmes no regressions in other tests because of these patches :)

Great work Stevel!

Thanks for looking into this and submitting all those patches. Drupal 7 will be better because of you!

Did the patches presented here get integrated into the current rc2 release?

Cheers, Lars.

Current state:

Schema API 39 passes, 0 fails, and 1 exception
Test statistics admin. 99 passes, 1 fail, and 0 exceptions
Basic upgrade path 50 passes, 12 fails, and 4 exceptions
Comment upgrade path 18 passes, 2 fails, and 0 exceptions
Filter format upgrade path 22 passes, 3 fails, and 1 exception
Locale upgrade path 119 passes, 8 fails, and 0 exceptions
Node body upgrade path 25 passes, 3 fails, and 0 exceptions
Poll upgrade path 206 passes, 2 fails, and 0 exceptions
Taxonomy upgrade path 1229 passes, 27 fails, and 0 exceptions
Upload upgrade path 18 passes, 2 fails, and 1 exception
Cancel account 207 passes, 14 fails, and 0 exceptions

The tests are now automatically run every day by http://drupal-ci.c--g.net/.

Select tests, subqueries 35 passes, 1 fail, and 0 exceptions

This is a broken test, over here: #1001242: DBTNG support for EXISTS conditions

SimpleTest functionality 89 passes, 21 fails, and 38 exceptions

Seems to be caused by the exception inComment fields

Update tests 40 passes, 1 fail, and 0 exceptions

Seems to be because PostgreSQL has a different logic to determining which rows count as having been updated to MySQL, does anyone know where there is an issue for this?

Differing row counts:
#805858: Affected rows inconsistent across database engines

Test run started: Tuesday, January 25, 2011 - 07:31

Test summary:
-------------

Comment fields 80 passes, 0 fails, and 1 exception
Comment interface 1363 passes, 0 fails, 0 exceptions, and 64 debug messages
Update tests 40 passes, 1 fail, and 0 exceptions
SimpleTest functionality 89 passes, 21 fails, and 38 exceptions

In summary,

#1008128: Do not use a single underscore as table and index separator on PostgreSQL and SQLite / #998898: Make sure that the identifiers are not more the 63 characters on PostgreSQL Will fix:

Comment fields 80 passes, 0 fails, and 1 exception

#805858: Affected rows inconsistent across database engines Will fix:

Update tests 40 passes, 1 fail, and 0 exceptions

#1008128: Do not use a single underscore as table and index separator on PostgreSQL and SQLite / #998898: Make sure that the identifiers are not more the 63 characters on PostgreSQL Will fix:

SimpleTest functionality 89 passes, 21 fails, and 38 exceptions

So just three issues until all tests pass!

#1003788: PostgreSQL: PDOException:Invalid text representation when attempting to load an entity with a string or non-scalar ID is not a fun bug at all.

sub

Meta issues must not be more severe than normal.

@linclark I'm fairly certain that as the referenced issues have not been resolved then this issue is still an issue.

I will re-test and let everyone know the results!

I've finally got around to testing PostgreSQL, and there are some failures, I've uploaded my jUnit results to a sandbox: http://drupal.org/sandbox/darthsteven/1811796 Most of them are in the upgrade path, it seems like Simpletest can create a simpletest environment within a simpletest environment on PostgreSQL.

I'll keep plugging away at things and try to clean up my Vagrant environment so I can share the test rig.

Core testing on Postgres needs to be enabled. Currently, only MySQL and SQLite are tested.

This issue should be also fixed for D7 in order to make PostgreSQL tests work.

Combined patches from #998898-95: Make sure that the identifiers are not more the 63 characters on PostgreSQL and #3262341-16: [D7] Database test table TEST_UPPERCASE causes PostgreSQL tests to fail to see if those two are enough to get tests running again in PostgreSQL.

Setting to NR just for the bot, we won't be committing a patch in this meta issue.

Good to see the test actually running all the way through.

12 failures doesn't seem too bad :)

We'll get the two issues stopping tests from running committed first then look at the remaining fails.

No-op patch to run tests.

@Liam Morland you beat me to it, thanks :)

Filled the first issue to fix remaining PostgreSQL failures: #3264314: [D7] DatabaseReservedKeywordTestCase and DatabaseTablePrefixTestCase using wrong Fully qualified table name in PostgreSQL

This one seems to be caused only by new tests introduced a year ago.

Adding another two issues where only tests are the problem (no core changes needed). I have added patches already:

#3264345: [D7] PollCreateTestCase - invalid input syntax for integer on PostgreSQL

#3264353: [D7] NodeCreationTestCase fails on PostgreSQL

Another noop to see what condition our condition is in.

As @poker10 mentioned in #3264314: [D7] DatabaseReservedKeywordTestCase and DatabaseTablePrefixTestCase using wrong Fully qualified table name in PostgreSQL some of the test results seem a bit inconsistent, but I think we can keep working our way through them for now.

Similar technique to what we did in #3081386: [META] Fully support PHP 7.4 in Drupal 7 to try and organise the noisy results a bit:

$ id='2319295' ; curl -s https://www.drupal.org/pift-ci-job/$id | grep -oE '(exception|fail): .* Line [0-9]* of .*:' | perl -pe 's#<.*?>##g' | sort | uniq -c | sort -rn
     11 fail: [Other] Line 774 of modules/file/tests/file.test:
      8 fail: [Other] Line 754 of modules/file/tests/file.test:
      6 fail: [Other] Line 801 of modules/file/tests/file.test:
      5 fail: [Other] Line 821 of modules/file/tests/file.test:
      5 fail: [Other] Line 816 of modules/file/tests/file.test:
      4 exception: [PDOException] Line 2256 of includes/bootstrap.inc:
      4 exception: [Notice] Line 396 of includes/ajax.inc:
      4 exception: [Notice] Line 2449 of modules/simpletest/drupal_web_test_case.php:
      3 fail: [Other] Line 259 of modules/simpletest/tests/upgrade/upgrade.test:
      3 exception: [Uncaught exception] Line 2284 of includes/database/database.inc:
      2 fail: [Other] Line 425 of scripts/run-tests.sh:
      2 fail: [Other] Line 258 of modules/simpletest/tests/upgrade/upgrade.test:
      2 fail: [Other] Line 236 of modules/simpletest/tests/upgrade/upgrade.test:
      2 exception: [Notice] Line 2450 of modules/simpletest/drupal_web_test_case.php:
      1 fail: [Other] Line 878 of modules/simpletest/tests/database_test.test:
      1 fail: [Other] Line 582 of modules/simpletest/tests/upgrade/upgrade.test:
      1 fail: [Other] Line 49 of modules/simpletest/tests/upgrade/upgrade.taxonomy.test:
      1 fail: [Other] Line 44 of modules/simpletest/tests/upgrade/update.aggregator.test:
      1 fail: [Other] Line 30 of modules/simpletest/tests/upgrade/upgrade.locale.test:
      1 fail: [Other] Line 29 of modules/simpletest/tests/upgrade/upgrade.node.test:
      1 fail: [Other] Line 1553 of modules/field/tests/field.test:
      1 fail: [Browser] Line 39 of modules/simpletest/tests/upgrade/upgrade.locale.test:
      1 fail: [Browser] Line 37 of modules/simpletest/tests/upgrade/upgrade.locale.test:
      1 fail: [Browser] Line 33 of modules/simpletest/tests/upgrade/upgrade.locale.test:
      1 exception: [Warning] Line 3856 of includes/bootstrap.inc:
      1 exception: [PDOException] Line 1252 of includes/bootstrap.inc:

Another child issue for what looks like backport of changes to the test only: #3264471: DatabaseUpdateTestCase::testExpressionUpdate assertion on number of affected rows fails in PostgreSQL

Added issue for file.test failures - #3264750: [D7] FileFieldWidgetTestCase::testMultiValuedWidget missing upload form causes fails on PostgreSQL

Another one which should be easy to fix (without touching the core): #3264826: [D7] FieldInfoTestCase::testFieldMap array order causes test fail on PostgreSQL

I have added the patch.

Next two issues only tests related - patches are attached (without changes to the core).

#3264845: [D7] AggregatorUpdatePathTestCase::testAggregatorUpdate - inconsistent array order causing test to fail on PostgreSQL

#3264866: [D7] BasicMinimalUpdatePath::testBasicMinimalUpdate is checking wrong key name on PostgreSQL

Combined patch of:

• 3264866-2.patch
• 3264826-5.patch
• 3264750-7.patch
• 3264471-17.patch

The test failures that remain seem to be pretty inconsistent. Let's try a couple of runs.

Adding 3265522-2.patch to skip the Upgrade (i.e. D6 -> D7) tests as they've been failing inconsistently with PostgreSQL and are arguably not so relevant 10 years on.

The test union fail in PHP 8.1 is another ORDER BY issue :) I will create a issue with patch for that.

It is very nice to see that otherwise all test should be passing after these patches.

Here is the issue with a patch for the last PHP 8.1 test failure on PostgreSQL: #3265579: [D7] DatabaseSelectTestCase::testUnion - inconsistent array order causing test failure on PostgreSQL

noop patch - I believe all the fixes are committed and core should now pass PostgreSQL tests with any luck.

I think we can finally call this one Fixed!

Thank you everybody that contributed.

We don't use PostgreSQL but watching you guys taking care of these has been amazing!