Comments

drunken monkey created an issue. See original summary.

drunken monkey’s picture

Status: Active » Needs review
StatusFileSize
new563 bytes
drunken monkey’s picture

Issue summary: View changes
drunken monkey’s picture

StatusFileSize
new841 bytes
new834 bytes

Seems the fail on Postgres is actually due to a Core bug: #1148856: Postgres schema doesn't support keylength on a unique index.
However, probably better to just change this for MySQL anyways, I guess? After all, it’s not 100% accurate and could, in very hypothetical scenarios, even lead to bugs (tasks not getting created since their data has the first 255 characters in common with another task’s).

diegopino’s picture

Yes, thanks this fix works.

But, personally, I feel using the whole data column as a key may be a design issue. Thinking loud here, it would be maybe better to apply an md5 hash on data itself, index that as an extra field/column and use that as unique to avoid double tracking. The hypothetical scenarios or something not being tracked because of a partial match do worry me specially because they would be quite hard to debug. I have always seen trackers like queues, that are by design totally unaware of any other entry. That makes them atomic and failsafe. Repeating an Solr document indexing less process intensive that querying the DB for its existence. Again, my 25cents and I came late to the discussion.

drunken monkey’s picture

Title: Fix test failures on MySQL » Fix current test failures
StatusFileSize
new570 bytes
new1.37 KB

First off, indexing doesn’t use our tasks system, so that’s completely independent from that.
The task types actually used by this module can be found in the three *TaskManager classes, and of all of those only the server task manager’s deleteItems type has a data column (in this case used for the deleted items’ IDs) that could conceivably (in my opinion) exceed 255 characters in length. And then for that to be falsely flagged as a duplicate there would need to be a second task with an identical length of the array of deleted items, identical item IDs for the first 255 characters but differences later on. (And do note that deleting all items for an index or datasource has its own task type. And that the task system will only be used if the initial attempt to just delete those items from the server right away failed for some reason.) I’d say that’s extremely far-fetched – and even if it should happen, the actual consequences would be minimal in most cases, as any undeleted items will just get deleted the next time any part of the Search API tries to load them.

So, I think we should just go ahead with this. You’re right, though, a problem here would be extremely difficult to debug. I’ll remind myself to put something about this in the release notes to ask for other opinions. If the majority ends up thinking that this is a bad move, we can also easily just drop the whole UNIQUE constraint again.

Unfortunately we now have a second test failure, this time against Drupal 8. Let’s fix both of them at once.
I’ll commit this once the test bot is happy.

drunken monkey’s picture

StatusFileSize
new846 bytes
new1.65 KB

Urgh.

drunken monkey’s picture

Status: Needs review » Fixed

Finally. Committed.

Thanks again for your input, DiegoPino!

  • drunken monkey committed 36225c8 on 8.x-1.x
    Issue #3228210 by drunken monkey, DiegoPino: Fixed current test failures...
diegopino’s picture

Sorry for my late response. Too many moving targets lately. Clever fix by identifying the driver! Thanks so much

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

grimreaper’s picture

Hello,

If this may help anyone. I encountered an issue when deploying on prod for my personal website. And this is what I had done.

Add in the my.cnf:

[mysqld]
innodb_file_format    = Barracuda
innodb_file_per_table = ON
innodb_large_prefix   = ON
innodb_default_row_format = 'DYNAMIC'

Restart MariaDB and then convert existing tables with:

mariadb --batch --skip-column-names --execute 'SELECT CONCAT("ALTER TABLE `", TABLE_SCHEMA,"`.`", TABLE_NAME, "` ROW_FORMAT=DYNAMIC;") AS MySQLCMD FROM information_schema.TABLES WHERE ENGINE="innodb" AND ROW_FORMAT != "DYNAMIC" AND ROW_FORMAT !="COMPRESSED" AND TABLE_SCHEMA="MYDRUPALDATABASE"' | mysql

Then the hook_update_N was ok.

wells’s picture

Issue summary: View changes
Related issues: +#3248297: Default tests failing

FYI I have opened #3248297: Default tests failing over at the Facets module which seems to be (somehow) related to this change. Tests that were passing as of 6 Oct. 2021 now appear to be failing with a similar key length error despite (because of?) this patch. I'm confused here about how this patch fixed an issue here but maybe causes the same issue in Facets. You can see in the failed test example on that issue the new UNIQUE KEY `task__unique` (`type`, `server_id`, `index_id`, `data`(255)) part of the query.

Anyone have thoughts on how these are related? I'm stumped at the moment.

mrshowerman’s picture

I seem to have the same issue as Grimreaper.

The update hook 8107 fails with this message:

General error: 1709 Index column size too large. The maximum column size is 767 bytes: ALTER TABLE "search_api_task" ADD UNIQUE KEY `task__unique` (`type`, `server_id`, `index_id`, `data`(255));

I assume it is due to the collation set to utf8_mb4, which leads to a necessary column size of 4 * 255 = 1020 bytes, exceeding the 767 bytes in our MariaDB 10.5.12 environment.

wells’s picture

It seems that the Facets tests failures and the issues noted by @Grimreaper and @mrshowerman are all the same thing — this commit made the module essentially incompatible with MySQL 5.6 in certain (default? and testbot) configurations.

With the deprecation of D8 maybe this is acceptable? Drupal 9+ requires MySQL 5.7+ where this doesn’t seem to be an issue.

mrshowerman’s picture

Thanks @wells for pointing me in the right direction. Turns out that back when I installed Search API, I was still using MariaDB 10.1.48, so the search_api_task table got created with the COMPACT row format. Even after upgrading to 10.5.12 later on, the row format didn't change, leading to the above error.
I fixed it by running the following:

MariaDB [dbname]> alter table search_api_task row_format=dynamic;

Now the update script was able to create the index.