I've spotted this issue while trying to vacuum sqlite3 database (it grew to over 440 Mb), while sqlite3_analyze reported only 20 Mb of data and over 95% of freelist. Upon vacum, sqlite3 reported `Error: no such collation sequence: NOCASE_UTF8`. Apparently NOCASE_UTF8 is custom collation for PDO driver, but it is invalid in sqlite3. Simple replacing NOCASE_UTF8 with NOCASE on db file solves the issue. This likely affects auto vacuum and in the result db file gets really huge with time.
More info here: https://bioinfoexpert.com/2019/02/28/investigate-reduce-the-size-of-drup...
Also reported in https://www.drupal.org/project/drupal/issues/2462175#comment-12991687

Comments

Anonymous’s picture

lpryszcz created an issue. See original summary.

toamit’s picture

I agree this is an issue, the db files are getting quite huge.

amateescu’s picture

Status: Active » Closed (works as designed)
Issue tags: -sqlite, -NOCASE, -COLLATE, -pdo, -collation

Apparently NOCASE_UTF8 is custom collation for PDO driver, but it is invalid in sqlite3.

Adding a custom collation is a feature of SQLite3 (https://www.sqlite.org/c3ref/create_collation.html) so it's perfectly valid to add our user-space NOCASE_UTF8 collation.

Since the SQLite database is generated and maintained by Drupal, you need to run the VACUUM command using Drupal's database layer in order for it to work properly.

toamit’s picture

I suspect the VACCUM command will fail. Any chance you could provide a drush command to accomplish this?
Please don't close the issue until its confirmed that VACUUM command will work when called via Drupal.

toamit’s picture

Status: Closed (works as designed) » Active
toamit’s picture

@amateescu the VACUUM command fails when called via drush, please see below

% cat command.txt 
VACUUM;

% `drush sql-connect` < command.txt 
Error: near line 1: no such collation sequence: NOCASE_UTF8
mmoulton’s picture

This still seems to be an issue as all of the drush sql:* commands simply execute shell commands using the 'driver' defined in the database config. They do not ever run the `Drupal\Core\Database\Driver\sqlite\Connection::open` command that is bootstrapping sqlite with all of the custom functions and the collation in question here?

While it seems the need to support custom functions and collations is very real, what is the best way to bring the bootstrapping of the driver out to the common management tools used maintain the state of our databases?

Is this a problem with the fundamental way the sql:* drush command work? If so, what can be done to bring continuity between core functionality of all the `Drupal\Core\Database\Driver` implementations through to the common management tools?

Obviously we can make some custom utils to get around this by using the native Connection:open, but that seems to miss the larger problem of a fundamental disconnect between the ubiquitous management tools, such as Drush, and Core.

andypost’s picture

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

pingwin4eg’s picture

Got this error today trying to manually edit a single field in phpstorm's database view, and trying to update the field in the query console:

[255] [UNKNOWN_ERROR] unknown error (no such collation sequence: NOCASE_UTF8)

So this problem really slows down a development process.

What are the solutions? How did you guys changed that NOCASE_UTF8 on an existing database?

pingwin4eg’s picture

Ok, I see that the core adds the collation for purpose:

    // Create a user-space case-insensitive collation with UTF-8 support.
    $pdo->sqliteCreateCollation('NOCASE_UTF8', ['Drupal\Component\Utility\Unicode', 'strcasecmp']);

So how to workaround this in other tools now?

pingwin4eg’s picture

Ok, spending a few hours I can't find any workaround to use phpstorm's DB tools with SQLite (except for viewing data in tables as is).

Have to use drush ev command like this: drush ev "\Drupal::database()->query('...')"

This way drush boots Drupal and executes a query in its context instead of bringing me directly to a DB console.

geek-merlin’s picture

While this is not a general solution, i ported SQLite Vacuum to D8.
Help appreciated with #3283573: Add a drush sqlite:vacuum command.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

apolitsin’s picture

Thanks @geek-merlin

```sh
drush php:eval "\Drupal\Core\Database\Database::getConnection('default', NULL)->query('vacuum;');"
```
https://git.drupalcode.org/project/sqlite_vacuum/-/blob/1.0.x/src/Comman...

Not being able to execute `vacuum` really creates a lot of problems.

At the same time the implementation fits in 1 line. Maybe this line should be added somewhere at the sqlite driver level to add such functionality?

wim leers’s picture

Priority: Normal » Major

Simply documenting #20 seems sufficient? 😊

(Many thanks, @apolitsin! 🙏)

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.