Problem/Motivation

As of #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols), migrate_drupal's integration tests for Drupal 7 started breaking. I looked into it and discovered that certain dump files (namely System.php) define 255-character keys, which exceed the 191-character limit imposed by InnoDB when using the utf8mb4 character set. This causes tests to fail because the system table can never be created.

Testbot didn't catch this because no Drupal 7 migrations/tests have actually been committed yet (one was RTBCed, which is what revealed this problem). The Drupal 6 dumps were unaffected because its System.php dump file defines the filename column as varchar_ascii, which circumvents this snafu.

Proposed Resolution

migrate-db.sh needs to export the MySQL character set when dumping tables. The attached patch sets the mysql_character_set key in the dump files, and includes regenerated versions of all the dumps (D6 and D7), which is why it's a big patch.

Remaining Tasks

Review and commit. This is holding up development of the D7 migration path, so it's high priority.

API / UI Changes

None.

CommentFileSizeAuthor
migrate_db_character_set.patch163.21 KBphenaproxima

Comments

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Issue summary: View changes
mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

I think an rtbc should come from someone who has been involved in the dump support - I'll just testify that the D6 upgrade process using migrate_upgrade module continues to work fine with this patch.

benjy’s picture

benjy commented

Why are there length changes in the dumps as well?

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented

I'm not 100% sure, but it looks like the Schema API removed support for specifying the length of int fields. I base this on the fact that, no matter how the fields are defined, int fields are always given a length of 11 by MySQL, and unsigned ints are always 10. It's consistent throughout the dump files in the patch. So if we commit, this should go away.

benjy’s picture

benjy commented
Status: Needs review » Reviewed & tested by the community

We discussed #4, and I think the noise of length changes will be a one off in this patch. Rest is all good.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Reviewed & tested by the community » Fixed

Committed 143d7b2 and pushed to 8.0.x. Thanks!

  • alexpott committed 143d7b2 on 8.0.x 
    Issue #2510072 by phenaproxima: UTF-8 support in MySQL driver breaks...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.