Problem/Motivation
Two client sites updated to the latest release of search_api today (1.21.0) and when running updb we get the following error:
> [notice] Update started: search_api_update_8107
> [error] Exception thrown while performing a schema update. SQLSTATE[HY000]: General error: 1709 Index column size too large. The maximum column size is 767 bytes: ALTER TABLE "search_api_task" ADD UNIQUE KEY `task__unique` (`type`, `server_id`, `index_id`, `data`(255)); Array
> (
> )
>
> [error] Update failed: search_api_update_8107
Steps to reproduce
- Update search_api from 1.20.0 to 1.21.0
- Run `drush updb`
| Comment | File | Size | Author |
|---|---|---|---|
| #75 | 3247781-74--remove_task_unique_index.patch | 2.28 KB | drunken monkey |
Issue fork search_api-3247781
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3247781-database-update-fails
changes, plain diff MR !20
Comments
Comment #2
todd zebert commentedYou beat me to posting this. I will add the update hook descriptions:
------------ ----------- --------------- ------------------------------------
Module Update ID Type Description
------------ ----------- --------------- ------------------------------------
search_api 8107 hook_update_n 8107 - Add a unique index to the
task entity type's storage.
search_api 8108 hook_update_n 8108 - Add configuration for boost
factors.
search_api 8109 hook_update_n 8109 - Enable index entity
references option by default on
existing indexes.
------------ ----------- --------------- ------------------------------------
Comment #3
cassioalmeida commentedCan confirm. Fixing the version on composer.json to 1.20.0 fixed for now.
Comment #4
joycehutch commentedI didn't have the error when updating my staging server, but it occured on my production site.
PRODUCTION:
20:13:24 ---------------- ----------- --------------- ---------------------------------
20:13:31 Module Update ID Type Description
20:13:31 ---------------- ----------- --------------- ---------------------------------
20:13:31 entity_browser 8201 hook_update_n 8201 - Updates entity browsers
20:13:31 to use the new media image
20:13:31 upload widget plugin.
20:13:31 search_api 8107 hook_update_n 8107 - Add a unique index to
20:13:31 the task entity type's storage.
20:13:31 search_api 8108 hook_update_n 8108 - Add configuration for
20:13:31 boost factors.
20:13:31 search_api 8109 hook_update_n 8109 - Enable index entity
20:13:31 references option by default on
20:13:31 existing indexes.
20:13:31 ---------------- ----------- --------------- ---------------------------------
20:13:31
20:13:31
20:13:31 // Do you wish to run the specified pending updates?: yes.
20:13:31
20:13:31 > [notice] Update started: search_api_update_8107
20:13:34 > [error] Exception thrown while performing a schema update. SQLSTATE[HY000]: General error: 1709 Index column size too large. The maximum column size is 767 bytes: ALTER TABLE "search_api_task" ADD UNIQUE KEY `task__unique` (`type`, `server_id`, `index_id`, `data`(255)); Array
20:13:34 > (
20:13:34 > )
20:13:34 >
20:13:34 > [error] Update failed: search_api_update_8107
20:13:34 > [notice] Update started: entity_browser_update_8201
20:13:34 > [notice] Update completed: entity_browser_update_8201
20:13:34 [error] Update aborted by: search_api_update_8107
20:13:39 [error] Finished performing updates.
STAGING:
17:35:03 ---------------- ----------- --------------- ---------------------------------
17:35:11 Module Update ID Type Description
17:35:11 ---------------- ----------- --------------- ---------------------------------
17:35:11 entity_browser 8201 hook_update_n 8201 - Updates entity browsers
17:35:11 to use the new media image
17:35:11 upload widget plugin.
17:35:11 search_api 8107 hook_update_n 8107 - Add a unique index to
17:35:11 the task entity type's storage.
17:35:11 search_api 8108 hook_update_n 8108 - Add configuration for
17:35:11 boost factors.
17:35:11 search_api 8109 hook_update_n 8109 - Enable index entity
17:35:11 references option by default on
17:35:11 existing indexes.
17:35:11 ---------------- ----------- --------------- ---------------------------------
17:35:11
17:35:11
17:35:11 // Do you wish to run the specified pending updates?: yes.
17:35:11
17:35:11 > [notice] Update started: search_api_update_8107
17:35:15 > [notice] Update completed: search_api_update_8107
17:35:15 > [notice] Update started: search_api_update_8108
17:35:15 > [notice] Update completed: search_api_update_8108
17:35:15 > [notice] Update started: entity_browser_update_8201
17:35:15 > [notice] Update completed: entity_browser_update_8201
17:35:15 > [notice] Update started: search_api_update_8109
17:35:15 > [notice] Enabled "Track changes in referenced entities" on all existing search indexes.
17:35:15 > [notice] Update completed: search_api_update_8109
17:35:15 [success] Finished performing updates.
Comment #5
laborouge commented+1
Comment #6
joycehutch commentedIf you look at the timestamps on my earlier message, the staging server updates succeeded, but when I moved the same code to production a few hours later, the database update failed. Subseqently, I copied the production database to a local site, and all three database updates worked. I then tried again on production, and the database update 8107 failed. So, I will drop back to search_api 1.20.
Comment #7
robert-io commentedI ran into this issue as well. After a little digging the issue lies (for me) in my MySQL configuration.
If you installed, like me, your site on MySQL <5.7:
By default MySQL uses the character set utf8 which means that we use 3 bytes for every 1 character. This means, column type of varchar(10) uses 30 bytes resulting in the max prefix size for compact row format to be equivalent to varchar(255). That is 255 * 3bytes = 765 bytes which is, two bytes less than the max of 767 bytes.
With innodb_large_prefix set to on and using row format COMPRESSED or DYNAMIC, you can increase the max prefix character size to 65536 bytes instead of 767 bytes. The below chart shows the max character length with InnoDB large prefix and [COMPRESSED| DYNAMIC] row formats. These values, expect for utf8mb4, are higher than the maximum row size of a table, so there is no way to hit these limits.
I've updated MySQL to 5.7, but my tables still had the COMPACT row format. You can check this by running "SHOW TABLE STATUS;". This will show the row format of your tables. If this is set to COMPACT, you have to update your tables to DYNAMIC.
You can check the default by running: "show global variables like 'innodb_default_row_format';". On <5.7 the default is COMPACT, so all tables will be created as COMPACT. In 5.7 the default is DYNAMIC.
https://community.pivotal.io/s/article/Apps-are-down-due-to-the-Maximum-...
I hope this helps!
Comment #8
maximpodorov commentedPostgreSQL 9.6 has similar problem:
Exception thrown while performing a schema update. SQLSTATE[54000]: Program limit exceeded: 7 ERROR: index row requires 16216 bytes, maximum size is 8191: ALTER TABLE {search_api_task} ADD CONSTRAINT search_api_task__task__unique__key UNIQUE (type,server_id,index_id,data);
Comment #9
ecj commented+1
alike #6 - can't update.
I also can not update my db version. neither settings.
falling back to version 1.20
Comment #10
ruuds commentedI also ran into this issue. Solved it by converting all tables from compact to dynamic, using the script on https://stackoverflow.com/a/22091156 which runs 'optimize table' on all tables in the database.
Always test on non-production first ;)
Comment #11
dorficus commentedI also ran into this issue and was able to get past it by running a query to alter the ROW_FORMAT of the
search_api_tasktable. For me, the query wasALTER TABLE search_api_task ROW_FORMAT=dynamic;, which I ran using SequelAce connected to my hosting provider.Once I ran that, I was able to run the database updates.
With that being said, I wonder if this should be patched to update
search_api_update_8107to include the query that changes the ROW_FORMAT if it is not set as dynamic.I would like the thoughts of others running into this issue before putting in the effort of writing a patch.
Comment #12
aleverenz commented#11 fixed this issue for me. Thanks!
Comment #13
ulfg commentedAlso same issue, had to add "set global innodb_large_prefix=on;" also. Tested on dev and staging environment
Commands used:
set global innodb_large_prefix=on;
ALTER TABLE search_api_task ROW_FORMAT=dynamic;
OPTIMIZE TABLE search_api_task;
Thanks for fixinfo!
Comment #14
maximpodorov commentedThese solutions are not universal and thus can't be the part of the module.
Comment #15
kdborg@gmail.com commentedI've found meeting the database requirements for Drupal 9 fixes this issue. The requirements are here: https://www.drupal.org/docs/understanding-drupal/how-drupal-9-was-made-a...
Comment #16
maximpodorov commentedUnfortunately, Search API is still ^8.8 || ^9.
Comment #17
damienmckennaFYI this is causing tests on other modules to fail, e.g. https://www.drupal.org/pift-ci-job/2234340
Comment #18
damienmckennaComment #21
taran2lAdding static patch so it can be used securely with composer patches
@DamienMcKenna please review
Comment #22
dorficus commented@maximpodorov That is disappointing that the fixes aren't universal, however I fully understand that only putting a fix for one DB type in when this could cause issues on multiple is not the best solution. For the time being, I'm going to recommend the query I added in #11 to my coworkers as we run into this issue or to not update the module just yet.
Comment #24
taran2lTests fails are unrelated to the issue. Back to needs review
P.S. It's weird that tests cannot be run when using MR
Comment #25
altrugon commentedI also confirm that #11 works. Thank you dorficus.
Comment #26
csbt23 commented#11 fixed the problem for me also! Big thanks
Comment #27
hs@henrikstrindberg.se commented#11 worked for me, thanks a lot!
Comment #28
e5sego commented#11 worked for me.
Comment #29
urix commented#11 fixed for me too!
Comment #30
mably commentedIn our case, we had to use these commands to make it work on our MySQL 5.5 instance:
logout & login (to get the global values)
Comment #31
hanoii#11 did fix it. What I did was create a dump of the database and then import it back in with the current defaults. That converted all tables to the new dynamic row format as and maybe other defaults as well.
Comment #32
mathilde_dumond commented#21 fixed the problem for us
Comment #33
berdirIt does fix the update yes, but a partial unique index really does seem weird. Are we sure that there is no risk that data is longer and the first N characters (be that 255 or 191) are the same? If that risk does not exist, then that would mean that there's no need for data to be longer anyway?
Comment #34
taran2lSo, limiting to 255 was not an issue - but limiting to 191 is? :)
Comment #35
dorficus commentedI should mention that an easier method if you're using the solution I posted in #11 is to just run the drush command
drush sqlq "ALTER TABLE search_api_task ROW_FORMAT=dynamic;"It's a little easier than trying to tunnel in using something like Sequel Ace, in my opinion.
Comment #36
drupixSame problems here... #35 worked for me.
Thanks!
Comment #37
gwvoigtEven with patch #21 I get:
Comment #38
dydave commentedHi everyone,
Thanks a lot for the feedback and help on this issue.
Reporting back after encountering the same issue with the following configuration:
Rolled-back for now to
1.20, until a clear solution appears, whether upgrading the DB system, patching or altering table.Thanks in advance!
Comment #39
rob230 commented#11 has worked for me as well.
Comment #40
cbUpgrading to MariaDB 10.4 worked for me.
For those on Pantheon, this can be done in your pantheon.yml like so;
Comment #41
luke adams commented+1 on #40 for the Pantheon peeps here.
Comment #42
damienmckennaFYI Pantheon announced today that later this month they are going to be providing a way of upgrading to MariaDB 10.4 from within the dashboard, but for now per #40 updating the pantheon.yml file is a relatively quick way of accomplishing this. A coworker mentioned that it can take a while for the database upgrade to run, so it's recommended to do that upgrade separate to any other deployments.
Comment #43
damienmckennaIMHO this should be a "won't fix" as it only affects Drupal 8 sites, D8 is EOL, and there's a workaround available (update the database), but it would be worth mentioning this in the release notes & project page.
Comment #44
berdirNo, this does not only affect Drupal 8, we're seeing this when updating D9 sites. That said, as commented in #33, a partial unique index on a field seems very strange, no matter how many characters that happens to be exactly.
Comment #45
taran2l@Berdir, yeah, but this is the MySQL limitation. Core is doing it automatically for regular indexes (but not unique), and the entire MySQL driver need an update, as newer MySQL versions can hold more than 191/255 chars. Anyway this is the whole different story.
See https://git.drupalcode.org/project/drupal/-/blob/9.3.x/core/lib/Drupal/C...
Comment #46
sleitner commentedThis bug has a history in core
Comment #47
dydave commentedAs per #37, patch #21 doesn't seem to fix the issue.
Back to Active.
Thanks!
Comment #48
nicolegalek commentedI am having a similar issue here - on D9 using pantheon. It is a new site build and I am unable to move the database from a dev to test environment due to the database error on search_api tables in this ticket. #40 did not work for me (I was already using 10.4).
Comment #49
taran2l@DYdave, #37 displays a totally unrelated fail, back to RTBC
Comment #50
dk-massive commentedI am experiencing this issue as well on Mariadb 10.4. Patch #21 does not resolve the issue for me.
Comment #51
hernani commentedWe ran into this issue. Solved it by converting all tables row format from compact to dynamic.
Comment #52
phannphong commented#11 solves the issue from my side. Thanks dorficus
Comment #53
mathiasgmeiner commented#21 worked for us
Comment #54
kevinquillen commentedThis happens on Pantheon. We cannot just convert the database at this time.
> IMHO this should be a "won't fix" as it only affects Drupal 8 sites, D8 is EOL, and there's a workaround available (update the database), but it would be worth mentioning this in the release notes & project page.
Not true - just updated to 9.3.0 and applied this update and saw it. Plus, we are currently on MariaDB 10.0 on Pantheon and cannot just upgrade it at this point in time.
The patch in #21 does allow us to deploy and update Pantheon successfully. Whether this is the right fix, I am not sure.
Comment #55
devad commentedPatch 21 didn't fix the issue in my case.
Maybe patch 21 is meant to work for new installations only? I have the working site and patching it didn't help.
@mathiasgmeiner did you do the patch testing with clean install or did you patch the site with already existing database? And what was your config?
-------------
#11 did fix my issue though. I am able to do the backup and local restore after applying the fix #11.
My config: D9.3.0, MySQL 5.7, PHP7.4
Comment #56
taran2lWhat exactly is not working? Could you provide more details? The idea is that you apply this patch and then re-run previously failed db updates
Comment #57
devad commentedI just realized after your latest post that the reason why the patch didn't work for me was because I didn't run the update.php after applying the patch. I didn't have problem with database update before (or I was not aware of it)... only with database restore... so my focus was on restore attempt.
Comment #58
devad commentedSorry for noise. Reverting the issue status back to where it was before my #55 change.
Thanks for your last comment @Taran2L.
Maybe it would be good to add one sentence into your comment where the patch #21 is - that it is needed to run database update after the patch is applied. So that others who have restore issues like me (or any other database issues with the same error message) are aware of it.
Comment #59
firfin commentedNeither The workaround in #11 or #13, nor the patch in #21 nor the combination of the these worked for me. Row style was already dynamic (see screenshot).
Anybody else have some solution?
Might be multiple problems here? I could create a separate issue if preferred.
Some background
I am trying to update an inherited drupal installation from D8 to D9. I don't feel it is wise to proceed without this update.
Output from ``drush updb``
The following updates are pending:
search_api module :
8107 - Add a unique index to the task entity type's storage.
8108 - Add configuration for boost factors.
8109 - Enable index entity references option by default on existing indexes.
Do you wish to run all pending updates? (y/n): y
Exception thrown while performing a schema update. SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 1000 bytes: ALTER TABLE {search_api_task} ADD UNIQUE KEY `task__unique` (`type`, [error]
`server_id`, `index_id`, `data`(191)); Array
(
)
Performing search_api_update_8107 [ok]
Failed: Exception thrown while performing a schema update. SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 1000 bytes: ALTER TABLE {search_api_task} ADD UNIQUE KEY `task__unique` (`type`, [error]
`server_id`, `index_id`, `data`(191)); Array
(
)
Comment #60
taran2lYou are using MyISAM engine, and it has a different limits, basically length of all fields is being used, i.e.:
However, running on MyISAM is not supported by Drupal, see https://www.drupal.org/node/2278745
Comment #61
joegraduateThis sounds like a major priority issue to me.
Comment #62
ruuds commentedEncountered the issue again on another server running MariaDB 10.1.45-MariaDB-0+deb9u1 Debian 9.12
Solved it by dropping the search_api_task table (which was empty), and recreating it as following, in which i removed the index and changed the charset from utf8mb4 to utf8.
After that running drush updatedb completed successfully.
I don't know the exact inner workings from the module, but:
1. We don't output data from the index directly in the website
2. We are migrating to a new server soon anyway.
I guess changing the charset might mess up the data in the index?
Comment #63
arnoldbird commented#62 worked for me.
Comment #64
pritam.tiwari commentedSolution #62 worked in my case.
Original values in create table query :
`data` longtext,
CHARSET=utf8mb4 ROW_FORMAT=DYNAMIC
New values as per #62 comment:
`data` longtext DEFAULT NULL,
CHARSET=utf8
Comment #65
drunken monkeyThanks a lot to everyone here who is exchanging information and trying to help resolve this. My apologies for not seeing this sooner – unfortunately, I’m way behind on the issue queues and with “Normal” priority there was no chance to see it sooner.
In cases where an update breaks the site, please feel free to a) set Priority to “Major” or even “Critical” (the latter especially if as many are affected as here) and b) contact me via Slack, mail or contact form if I don’t react within a week or so. (If it’s critical, I guess you can even ping me right away, to make sure I see it ASAP.)
@ NickDJM: Thanks a lot for reporting this issue!
@ Taran2L: Thanks a lot for the patch, looks great!
As it should solve the problem for a lot of people, and there is no clear way (as far as I can see) to fix the problem for Postgres as well, I already went ahead and committed it (with minimal style changes). Thanks again!
Regarding the problem testing MRs, yes, that’s pretty annoying. See #3190024: Problem with test dependencies when testing issue forks – unfortunately, no-one on the d.o dev team has seemed interested in providing support for it so far.
@ joegraduate: Thanks a lot for switching this ticket’s priority to “Major”. That way, I saw it at least in time to fix this before the next (1.22) release.
@ Berdir: You’re right, a
UNIQUEkey with a prefix length is pretty strange. I was worried about that, too.While it seems pretty unlikely that this would ever actually lead to a problem, and even then it shouldn’t be a big one (as the task system isn’t critical to normal operations), it’s still an undesirable state of affairs.
@ Postgres users: Sorry that this still won’t work for you currently (for some setups, at least), but for Postgres I have even less an idea of how to fix this. (Unfortunately, Postgres doesn’t even support prefix lengths for keys – see #1148856: Postgres schema doesn't support keylength on a unique index. (Btw, I’d appreciate it if someone could help me finally get this committed!))
Maybe we should just get rid of that
UNIQUEkey completely? It’s really just a safety net, the other changes in #3193690-10: Duplicated task records added into the search_api_task table should already be enough to ensure no duplicate tasks are added. And if it causes this many problems, then I don’t think it’s worth the trouble.@ everyone: Who’s in favor of just removing that
UNIQUEkey? Does anyone see problems with that?Comment #67
taran2l@drunken monkey,
afaik, just add them a require-dev in your composer.json, let's open a follow-up to figure it out .. I'm almost sure that I was able to run tests with forks ...
I've checked it one more time and I see a circular dependency issue: search_api requires search_api_autocomplete which requires search_api, but in 1.x version, however the forked version will be on a random branch, see
I guess this has changed since with Composer 2 intro ... let's maybe create a follow up
And THANKS for finally committing this
Comment #68
drunken monkeyOK, of course I was a bit too hasty and even created a release with this commit before finally remembering that there are no automatic entity schema updates anymore.
So, this would actually have needed an update hook to work correctly. The impact is minimal, though, mostly just the annoying error on the status report – and only for people who had already updated to 1.21 successfully before.
So, I think we should probably first figure out how to proceed here before supplying that update hook (and maybe then just creating a quick and small new release).
@ Taran2L: Wow, thanks a lot for figuring that out, that’s the first helpful insight I received for the MR problem!
Instead of creating a new issue, though, you can just reply in the existing one – if it turns out we can resolve this within the Search API (or Search API Autocomplete) module, we can still move the issue.
Comment #69
upchuk commented@drunken monkey yeah :) we just noticed our builds failing as we have coverage to ensure we never have entity definitions that need changing.
What would be the problem with providing the update hook in a patch release (or a regular release cause it's not yet using semver)?
Thanks
Comment #70
devad commentedI would like to add that the https://www.drupal.org/project/devel_entity_updates module helped in this case.
I have installed it and run "drush entup" afterwards - and the status warning about the "search_entity_task" entity witch needs an update is fixed.
Whatever code snippet this module is using during the "drush entup" command can be used probably for our update hook needed here.
Comment #71
drunken monkeyNone at all. However, I think we should first decide whether the
UNIQUEkey should just be dropped again, in order to not have the same problem right after the release again, and be in need of a third release within weeks.Do you have any opinion on that?
Comment #72
rob230 commentedAre there going to be any issues with 1.22 for people who altered the table to dynamic? Or does it not matter?
Comment #73
steveoriolI have the same error on all the websites where I have updated to 1.22 :

Comment #74
drunken monkeyOK, if no-one else has an opinion on this, and since I’d like to resolve this (and create a new release without that problem) ASAP, I suggest just dropping that problematic
UNIQUEkey again. The tiny benefit doesn’t seem to compare well to the numerous problems it has caused. Also, it seems the cleanest solution, which will work for all DBMSs.Please review/test and comment on the attached patch! If it works for everyone and no-one objects within the next few days, I’ll commit it.
(Please run the database updates after applying the patch. Afterwards, the error on the “Status reports” page should vanish.)
I don’t think so, no. But I haven’t checked.
Comment #75
drunken monkeyComment #76
drunken monkeyComment #78
steveoriolHi @drunken, I confirm that your patch works very well, thank you :-)
Comment #79
rgristroph commentedI ran into this, and the command in #11 worked.
It's a bit of a frustrating type bug, because I think depending on various factors, it can potentially not show up on lower environments and then raise it's head on prod, if the prod DB was created when your hosting had an older version of mysql, and then if mysql got updated everywhere the DBs might have been loaded fresh from a dump on all the lower environments while that might not be the case on prod.
Comment #80
anybody+1 for #75, thank you very much for your work on this @drunken monkey!
Comment #81
shaunlaws commented#75 got rid of the error on the status report. I reindexed my site and ran a few test searches. Everything looked good.
Comment #82
nick hope commented#75 works cleanly for me too. I re-indexed for the sake of it and search still seems to be working fine. Thank you @drunken monkey.
Comment #83
anybodyComment #85
drunken monkeyThanks for all the feedback, good to hear this works. Then let’s just do it like that.
Committed and created both a change record and a new patch release.
Thanks again, everyone!
Comment #87
drunken monkeySorry for bumping this again, but it seems this solution still doesn’t work for everyone: #3260230-6: Error on search_api_update_8110.
Also, I thought it would probably make sense to not even add that index in the first place in case someone updates from an earlier module version?
In any case, it would be very helpful to get some people to test or look at that patch before committing, so we don’t introduce yet another bug in connection with this cursed index. (However, as the patch just changes the existing
search_api_update_8110()function, testing only makes sense for people who have not yet installed that update – or, even better, not even update #8107.)Comment #89
jadowd commentedHey There all,
Having encountered this problem on Acquia Hosting with a client, we discovered that simply un-installing the module, then reinstalling it, will address the issue. The problem is the database index tables that were constructed under an older version of mysql (or percona) which allowed for some construction of problematic indexes that are no longer permitted. Trying to update the module will clash with the database schema prescription, however, uninstalling, then reinstalling it, will re-create the table (actually, and obviously, all of the relevant tables) under the prescription of the updated DB schema.
I hope that helps.
Thanks,