Problem/Motivation
CR https://www.drupal.org/node/2510940 introduced a limitation on index length because of database limitations at the time.
These limitations are long gone from the minimum supported engine versions, as they have both moved on to use the DYNAMIC row format as the default. However, you can still run into the limitation if someone manually changed their tables to use the COMPACT row format.
Drupal still adds the 191 length limitation on all indexes, though, even when most sites no longer have said limitation. So if someone adds an index on a 'list_string' or other varchar field with default length of 255, it actually cuts the indexed field to a lower size, making it not function. It even makes matters worse, because an index that can never cover the requested data will be consulted for no reason, slowing the query down more than if there were no index at all.
Because this happens silently, people may think they are improving their database performance by adding an index, but are actually making it worse.
Steps to reproduce
Add an index on a list_string base field and see it cut to 191 characters, killing performance.
Proposed resolution
Either refactor Schema::getNormalizedIndexes() to run the following query from the docs and then only add the limit if the row format is COMPACT or REDUNDANT:
mysql> SELECT NAME, ROW_FORMAT FROM INFORMATION_SCHEMA.INNODB_TABLES WHERE NAME='test1/t1';
+----------+------------+
| NAME | ROW_FORMAT |
+----------+------------+
| test1/t1 | Dynamic |
+----------+------------+
Or, do not add the 191 character limit at all unless a setting is specified where you opt in to this behavior.
Remaining tasks
Decide what to do and do it
User interface changes
Introduced terminology
API changes
Data model changes
Index size may increase, for the better.
Release notes snippet
Issue fork drupal-3620372
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
daffie commentedYes, we can switch from COMPACT to DYNAMIC rows for MySQL and MariaDB.
Both the minimum required version for MySQL and MariaDB in D11 are enough.
The SQL query to do that is:
The problem is that every table that needs to make that change will do a full table rebuild.
It will mostly effect older Drupal sites, that were created with older versions of MySQL and MariaDB.
Writing the hook_upgrade and making the other changes in core will not be the big problem.
Whether to put site owners through the change/migration of converting every table from ROW_FORMAT=COMPACT to DYNAMIC is the question.
So, how much performance/new features can we get in Drupal to make it worth for sites owners to do the upgrade?
@kristiaanvde: Can you update the issue summary with all the great features we can add to Drupal or how much performance we would gain.
Comment #3
kristiaanvandeneyndeHold up, I am not advocating for migrating every database that is on COMPACT or REDUNDANT to DYNAMIC without people's consent :D They may very well have had a good reason to manually set them that way (or keep them that way if legacy DB).
What I'm saying is that any new Drupal site is installed on a DB version that out of the box uses DYNAMIC. And DYNAMIC has no such limitation we still see in Schema::getNormalizedIndexes().
So instead of using a bazooka here and ruining indexes for ALL databases, why don't we only still set the 191 character limit when we know that the row format is COMPACT or REDUNDANT?
Either that, or never set the limit anymore unless a setting is specified that opts into the limit. Then legacy DBs or people who manually specify a COMPACT row format and want to create such a problematic index might get a DB error until they turn the setting on.
We can gain performance because we no longer create indexes that do nothing, but will be consulted. In my metrics, a lookup without an index was 160ms, with a fixed index was 100ms and with the broken index Drupal adds it was >300ms.
Comment #4
daffie commentedI am working on this.