I've just tried brand new Drupal 8 beta1, and for sake for interest, installed it using SQLite database in Kubuntu 14.04 amd64 laptop.

Installation took.. quite a bit. During installation I've noticed that .ht.sqlite has "-joural" file nearby which is created/deleted, so it's rollback journal with journal_mode=DELETE (the default).

I've made small change in core/lib/Drupal/Core/Database/Driver/sqlite/Connection.php, to force WAL journal mode:

--- Connection.php      2014-10-01 12:54:04.175647920 +0300
*************** class Connection extends DatabaseConnect
*** 127,132 ****
--- 127,133 ----
      $pdo->sqliteCreateFunction('rand', array(__CLASS__, 'sqlFunctionRand'));
      $pdo->sqliteCreateFunction('regexp', array(__CLASS__, 'sqlFunctionRegexp'));
  
+     $connection_options['init_commands'] = array('wal' => 'PRAGMA journal_mode=wal');
      // Execute sqlite init_commands.
      if (isset($connection_options['init_commands'])) {
        $pdo->exec(implode('; ', $connection_options['init_commands']));

Now installation is measurably faster.

Using unmodified Drupal source, time of core Drupal modules installation (since "Save and continue" in Database configuration page and until 100% of module installation is reached and loading indicator for next page in browser tab appears) is:
~6m:55s

Using journal_mode=wal:
~2m:46s

I can hear how sparse my laptop HDD drive rattling is in WAL mode, compared to non-stop noise in default journal mode.

It would be nice if upon new installation, if current available PHP SQLite extension supports WAL (since version 3.7.0), that settings.php would be generated in a way that journal_mode=wal would be enabled upon connection.

Possibly, there could be opt-out "Use WAL journal mode" checkbox in "Advanced" fieldset in Drupal installation database settings page, or a select box of all available journal modes, if that felt to be necessary (though more advance/exotic settings/PRAGMAs could be provided in settings.php manually after all).

I do not know if there are some significant drawbacks using this mode in PHP, Drupal, or for web app as whole, please comment if any.

The advantages of Write-Ahead Logging (WAL)

  1. WAL is significantly faster in most scenarios.
  2. WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently.
  3. Disk I/O operations tends to be more sequential using WAL.
  4. WAL uses many fewer fsync() operations and is thus less vulnerable to problems on systems where the fsync() system call is broken.
CommentFileSizeAuthor
#3 2348137-3.patch1.09 KBdaffie

Comments

daffie’s picture

Version: 8.0.0-beta1 » 8.2.x-dev
Priority: Normal » Major

The minimum SQLite version is now 3.7.11 (#2605594: Change minimum required sqlite version to 3.7.11). So now we can do this!

I think this is important. So I have set the priority to major.

daffie’s picture

Component: database system » sqlite db driver

Moving this to the SQLite driver.

daffie’s picture

Status: Active » Needs review
StatusFileSize
new1.09 KB
daffie’s picture

Issue summary: View changes
daffie’s picture

The testbot is a lot faster with WAL enabled:

  1. SQLite with PHP 5.5 from 33 minutes to 30 minutes.
  2. SQLite with PHP 7 from 19 minutes to 12 minutes.
daffie’s picture

Title: Enable WAL journal mode by default for SQLite database upon installation if it's supported » Enable WAL journal mode by default for SQLite database

The WAL journal mode is available on SQLite as of version 3.7.0. Our minimum required version for SQLite is 3.7.11. So WAL journal mode is available by default.

andypost’s picture

Issue tags: +Needs change record

As we raised minimum version already it makes a lot of sense
Also requirements page should be updated with suggested version

Change record should point about perf increase and new suggestion

daffie’s picture

Issue tags: -Needs change record

Change record added.
The requirements page was already updated.

andypost’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: +Performance

Works great

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs change record updates, +8.2.0 release notes

https://www.sqlite.org/wal.html lists the drawbacks - the top two seem significant.

  1. WAL normally requires that the VFS support shared-memory primitives. (Exception: WAL without shared memory) The built-in unix and windows VFSes support this but third-party extension VFSes for custom operating systems might not.
  2. All processes using a database must be on the same host computer; WAL does not work over a network filesystem.

I think the CR at least needs to be updated to mention the second one and how to disable it.

daffie’s picture

Status: Needs work » Reviewed & tested by the community
Issue tags: -Needs change record updates

Updated the change record with the wishes of alexpott.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed d317f6b and pushed to 8.2.x. Thanks!

  • alexpott committed d317f6b on 8.2.x
    Issue #2348137 by daffie: Enable WAL journal mode by default for SQLite...
Talkless’s picture

I'm very happy to see it implemented! Thanks!

daffie’s picture

@Talkless: Can you start testing Drupal 8.2.x (HEAD) and post issues for any bugs that you find.

  • alexpott committed d317f6b on 8.3.x
    Issue #2348137 by daffie: Enable WAL journal mode by default for SQLite...

  • alexpott committed d317f6b on 8.3.x
    Issue #2348137 by daffie: Enable WAL journal mode by default for SQLite...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.