Drupal\system\Tests\Database tests + closely related tests do not pass on SQLite.
For multiple reasons, in prioritized order:
-
SQLite: StatementPrefetch is not used and dead code due to logic error.
sqlite\ConnectionextendsDatabase\Connection. Its constructor intends to override$statementClass, but only does so after calling the parent constructor. The parent constructor immediately sets thePDO::ATTR_STATEMENT_CLASSattribute on the connection.In turn, SQLite uses the regular
Drupal\Core\Database\Statementclass. None of the custom SQLite code - includingStatementPrefetch- is ever called.In addition, the former
StatementPrefetchapproach is no longer compatible with PHP core. PDO immediately throws an exception stating that a "user-supplied statement class must derive from\PDOStatement."However, #2304461: KernelTestBaseTNG™ proves that the workaround is no longer necessary and obsolete. SQLite works perfectly fine with the regular
Statementclass.The only workaround that is still required is the expansion of numeric values in the SQL statement, since http://bugs.php.net/bug.php?id=45259 is still unresolved.
But that can be easily resolved by moving the former code from
Statement::getStatement()into a driver-specific override ofConnection::expandArguments().→ Expand the arguments in the SQLite
Connectionclass already. Remove all other workarounds (dead code).New coverage is added to Database tests.
-
SQLite
Insertquery does not account forINSERT FROM ... SELECT.The driver-specific optimization for
DEFAULT VALUESincorrectly kicks-in, because$this->insertFieldsis empty.→ Fix
Insertquery to account for a non-empty$this->fromQuery.Covered by
InsertTest. -
SQLite
REGEXPuser function exists, but is wrongly implemented.The current user function was implemented after SQLite's native
regexp(subject, pattern)function, but thestring REGEXP patternSQL operator results inregexp(pattern, subject)(reversed arguments).→ Fix the arguments.
Covered by SelectTest::testRegexCondition().
-
SQLite does not interpret
'\_'as a string literal inLIKEwithout anESCAPE '\'postfix.At the same time, MySQL breaks with
LIKE ... ESCAPE '\'.All
SELECTqueries usingLIKEare required to use the Select query builder, which handles the situation viaConnection::mapConditionOperator()already.However,
Config\DatabaseStorage::listAll()does not use the Select query builder.→ Use a proper Select query builder in
Config\DatabaseStorage::listAll().Covered by Database
BasicSyntaxTest+ ConfigDatabaseStorageTest. -
SQLite does not natively support
CONCAT_WS().Used by Views, also in D8.
→
CONCAT_WS()is not ANSI SQL, but is supported by MySQL + PostgreSQL, and can be easily supported in SQLite.New coverage is added to Database tests.
-
SQLite's date/calendar handling is limited.
SQLite does not support advanced date/calendar operations (such as identifying 2000-01-01 as the 52nd week of 1999).
→ The affected test of Views won't ever pass on SQLite and needs to be wrapped into a database driver condition.
Proposed solution
- Make qa.drupal.org testbots additionally execute all Database tests against SQLite when testing Drupal core.
- Add the corresponding fixes.
Why the changes to run-tests.sh?
-
The approach was discussed in recent meetings for modernizing the testbot already.
-
It's the
Databasecomponent that claims to support multiple backends, so proving that claim means to run only theDatabasetests against more than one backend — a.k.a. explicit test coverage. (1 min 13 sec) -
run-tests.shis adjusted to check whether it is executed on a qa.drupal.org/PIFR testbot and asked to run the full Drupal core test suite (--all). -
Only if both conditions are met,
Databasetests are executed against SQLite prior to the actual core test suite (which still runs against MySQL). -
Test failures in SQLite do NOT affect the final test result. For now, test results are only visible in the test review log and not recorded (i.e.,
--keep-resultsis explicitly disabled).
Major milestone.
This is how backend-specific component testing is supposed to work and how it's done everywhere else. That's why the application should have total control over the CI test/build script and why the same approach is found in Travis-CI (working example).
Once we fully migrated to PHPUnit, the executed operation will look like this:
$ DRUPAL_DATABASE=sqlite://localhost/build/db.sqlite phpunit -c core --group Database
(Unlike run-tests.sh, this will run all tests tagged with @group Database; i.e., including Database backend tests of other components.)
Why a single issue?
As soon as you swap out the Database driver, all errors are exposed.
| Comment | File | Size | Author |
|---|---|---|---|
| #53 | interdiff-2318191-47-53.txt | 0 bytes | babruix |
| #53 | 2318191-53.patch | 41.84 KB | babruix |
| #47 | sqlite.47.patch | 41.87 KB | cilefen |
| #47 | interdiff-45-47.txt | 404 bytes | cilefen |
| #45 | sqlite.45.patch | 41.35 KB | cilefen |
Comments
Comment #1
sunThe initial patch should fail (tested locally).
Attached patch should pass. (still contains the temporary change to Database tests)
Comment #6
sunLooks like MySQL isn't happy about
LIKE ... ESCAPE '\'.Comment #7
sunUpdating issue summary accordingly.
Comment #9
sunOnce more, complete set.
Comment #10
sunComment #12
sunComment #17
dcrocks commentedI use SQLite all the time and this patch appears to fix at least one of my problems(#2267563: Uncaught SQLite failure breaks responsive images). I don't know what the exact failure was, or what in this patch fixes it, but the failing behavior I was seeing stopped after applying this patch.
Comment #18
sunNew approach, single patch, full SQLite test coverage (on testbots).
Comment #19
sunAdjusting proposed solution in issue summary accordingly.
Comment #20
sunAnd we have real results:
Comment #21
dcrocks commentedCreated new clone of D8.0, applied patch, and installed. Everything went cleanly. Tried responsive images and everything works properly. Only thing missing is a code review, else I think this is RTBC.
Comment #22
sunFixed
DatabaseExceptionWrapperTestto account for MySQL/SQLite differences (see new/last commit in serial).Clarified
run-tests.shchanges in issue summary.This should fully pass against both backends and thus RTBC.
Comment #23
sunPassed, as promised:
Comment #24
dcrocks commentedI would like to see this go in sooner rather than later.
Comment #25
sunComment #27
sunRebased.
Comment #28
dcrocks commented#27 installed cleanly on a new D8.0 clone and still solves my problem.
Comment #29
webchickHm. There's a lot going on in this patch. I would be more comfortable committing it if we could isolate it to only the changes required to get DB tests passing on SQLite, and boot at least:
- Make qa.drupal.org testbots additionally execute all Database tests against SQLite when testing Drupal core.
- In addition, the former StatementPrefetch approach is no longer compatible with PHP core.
...to separate sub-issues to discuss/commit on their own. Removing core/lib/Drupal/Core/Database/StatementPrefetch.php probably needs eyeballing from at least Larry, and that drastic of a change to how testbot tests core needs eyeballing from probably lots of people.
Comment #30
dcrocks commentedThis patch also fixes #2275251: Display mode management broken with sqlite.
Comment #31
attiks commented#29 This patch solves a lot of problems, so it would be nice to get this in soon.
Comment #33
dcrocks commentedGiven there are uncaught errors in D8 without this, I don't see how D8 can go beta.
Comment #35
dcrocks commentedI'd like to make this critical but more importantly it needs attention. I don't see how D8 can go beta with uncaught failures in the SQlite environment.
Comment #36
dcrocks commentedThis will also fix #2342191: Subtle bug in Drupal\Core\Database\StatementPrefetch::current().
Comment #37
dcrocks commented#2349789: Responsive Image Mappings are not listed with sqlite has a patch for DatabaseStorage.php that is almost identical to the one in this patch that addresses my issue (#2267563] as well. Given there has been no progress here, and that a re-roll is needed here in any case, can someone review that issue's patch and possibly we can get it committed more quickly. Given that D8.0 beta has uncaught data failures with SQLite I'd like to mark this as critical but I somehow feel that would get stomped on rapidly.
Comment #38
dcrocks commented#2349789: Responsive Image Mappings are not listed with sqlite has been committed.
Comment #39
tstoecklerBecause of 4. in the issue summary Sqlite does not work *at all* for config names with an underscore. Try that with a module like "file_entity"...
This is certainly critical as it renders the site *heavily* unusable.
Edit: ahh that was fixed with #2349789: Responsive Image Mappings are not listed with sqlite. still I think this should be critical
Comment #40
cilefen commentedReroll of #27.
Comment #44
dcrocks commentedI can't find UnitTestBase anywhere in drupal 8. Shouldn't
remain the same in "DatabaseExceptionWrapperTest.php"?
Comment #45
cilefen commentedThank you @dcrocks.
Comment #47
cilefen commentedComment #50
dcrocks commentedGot a current clone of D8, patch installed cleanly, installed D8 on OS X 10.10 without any errors. Will try other tests soon.
Comment #51
chx commented> the workaround is no longer necessary and obsolete. SQLite works perfectly fine with the regular Statement class.
[citation needed]. Brief testing showed this to be correct but I do not think we should dismiss this easily. Reminder:
I look at https://github.com/php/php-src/blame/master/ext/pdo_sqlite/sqlite_statem... and on and I see zero changes in a decade or so to the functionality of this driver. Did SQLite change behaviors? WAL doesn't seem to be the default. I am a bit baffled about what's changed and when. I will ask Damien to look at the issue.
Edit: Damien claims my testing was incorrect.
Comment #52
chx commentedHere's a test script
it ends. I tried many inserts just to try to kick the optimizer out but no success.
Comment #53
babruix commentedRe-rolled.
Comment #54
webchickWe had a branch maintainer call and agreed to keep this critical. Unlike the PostgreSQL issues, we actually need SQLite for various things (e.g. testbot) so we do need to fix this.
However, it'd probably be way easier to make faster progress if the individual issues were split out into finite sub-issues each on one thing. So turning this one into a "meta" issue.
Also adding related infrastructure issues that will help ensure we don't keep introducing these regressions in the future.
Comment #55
star-szrComment #56
daffie commentedCreated #2427311: SQLite does not natively support CONCAT_WS() a sub-issue for
CONCAT_WS.Comment #57
daffie commentedCreated #2427773: SQLite REGEXP user function exists, but is wrongly implemented. a sub-issue for
REGEXP.Comment #58
amateescu commentedLet's mark the sub-issues major, there's no point in opening a bunch of criticals just because the patch here is too big.
Comment #59
daffie commentedPart 4 from the issue summary no longer applies. It is fixed in #2349789: Responsive Image Mappings are not listed with sqlite. If I am wrong please let me know.
Comment #60
daffie commentedCreated #2427875: SQLite Insert query does not account for INSERT FROM ... SELECT. a sub-issue for
INSERT FROM ... SELECT.Comment #61
daffie commentedCreated #2428377: SQLite Schema class is missing use Drupal\Component\Utility\Unicode a sub-issue for Schema class.
Comment #62
daffie commentedCreated #2428695: SQLite date handling is wrongly implemented and arguments handling needs override a sub-issue for date and arguments handling.
Comment #63
daffie commentedIf you apply the patch from the open sub-issues and you run the command:
php ./scripts/run-tests.sh --dburl sqlite://localhost/sqlite/db.sqlite DatabaseAll the tests passes.
Comment #64
daffie commentedIn the patch there is also an update to the run-tests.sh script for adding the SQLite database tests to the script. Is that something we want to add in this issue or move to a followup issue or do not do that at all?
Comment #65
star-szrSetting to active since it seems the patch on this issue is being chunked out, and usually metas stay at active until resolved.
Edit: Also woah, lots of action on the sub-issues. Thanks @daffie!
Comment #66
dawehnerLet's be honest, we should fix all failures, not just the database related ones.
Comment #67
bzrudi71 commented@dawehner asked for a SQLite testbot, here we go: temporary SQLite Bot. Happy hacking :-)
Comment #68
webchickHey, awesome!! Thanks so much for that, bzrudi71!
Comment #69
amateescu commentedI think I would prefer to keep the original scope of this meta, mostly because the existing issue summary was carefully planned and written and I wouldn't want to drown it in a sea of "fix everything" issues.
Opened #2454513: [meta] Make Drupal 8 work with SQLite to track all the other tests that need to be fixed on the temporary SQLite testbot.
@bzrudi71, I'm wondering which script is used to run the tests on that temporary bot. I assume it's not core's
run-tests.shbecause that testbot is running into the same problem that was fixed a few months ago: #2343389: Drupal core testing fails to write results because of SQLITE limitationComment #70
dawehner@amateescu
So now every database test is green, doesn't that mean we can close this issue?
Comment #71
amateescu commentedThere's still #2428695: SQLite date handling is wrongly implemented and arguments handling needs override open, but it's RTBC now so I'd say it's ok to close this :)