Support for utf8mb4

Would be nice to have update to this as an option on migrate/clone.

Yes, we have to support it somehow.

Steps:

1. add lines to my.cnf
2. add db settings to settings.php, see patch
3. convert database, run drush utf8mb4-convert-databases

We currently don't control the my.cnf ... so that part is up to the sysadmin.

Converting databases might be risky to fully automate ... we could create a task for it which could then be run via e.g. Views bulk operations. But I'm not sure it's worth the effort. Unless we find someone to do or sponsor it.

As we don't want to add the extra connection details to pre Drupal 7.50 sites we'd have to add a check. Here's a work in progress ... This currently fails on Error: Class 'Database' not found in...

Just in case anyone sees this warning output from Drush:

4 byte UTF-8 for mysql is disabled. See the documentation on adding 4 byte UTF-8 support for more information. (Currently using Database 4 byte UTF-8 support Disabled)

It shouldn't be a warning, and has already been fixed upstream (but will take time to trickle down). See There is no documented way to suppress warnings about missing 4 byte UTF-8 support.

I've for now simplified the to check to a drush_get_option() call.

Add to e.g. /etc/drush/drushrc.php

$options['utf8mb4IsSupported'] = TRUE;

It works for a fresh install of Drupal 7.50. Sites migrated to 7.50 without database converted also works.

After a verify task, site <7.50 throws an exception PDOException with message:
Syntax error or access violation: 1253 COLLATION 'utf8mb4_general_ci' is not valid for CHARACTER SET 'utf8'

To solve this issue, we can check if Drupal version supports utf8mb4:

if (method_exists($connection, 'utf8mb4IsSupported') && drush_get_option('utf8mb4IsSupported', FALSE)) {
  $this->data['utf8mb4IsSupported'] = TRUE;
}

The difficulty there is that during the verify task Drupal is not bootstrapped far enough to have $connection available.

We have package info in our frontend database, but not in the drush context about a platform.

An alternative would be to detect the version number of the Drupal installation.

Thanks that check works well.

I think we can even remove the drush_get_option('utf8mb4IsSupported') call, as Drupal 7.50+ will work OK anyway. It'll just mention on admin/reports/status that your db is not converted yet.

Yes, you're right. Patch updated.

Thanks, committed.

:( Install now fails

Exception Object [notice]
(
[message:protected] => Database 4 byte UTF-8 support: Not
supported

http://ci.aegirproject.org/job/P_Aegir_Puppet_Module_functional_test_Aeg...

I think we have to use the extra drush_get_option anyway. I've added it for now, making Jenkins happy http://ci.aegirproject.org/job/P_Aegir_Puppet_Module_functional_test_Aeg...

Maybe we can create a property on the db server class to indicate if a server supports it.

Yes, we could reuse the code from DatabaseConnection_mysql::utf8mb4IsSupported to automatically set the db server class property on provision-save.

Check database 4 byte UTF-8 support for Drupal 7 and create utf8mb4_is_supported property in db service class.

Thanks, looks good. Committed.

AFAICS the function utf8mb4_is_supported() will never return TRUE on any database that uses InnoDB. InnoDB's max key length is 767 bytes, utf8mb4 uses up to 4 bytes per codepoint, so the maximum CHAR/VARCHAR key length is 191. Subsequently, the CREATE TABLE statement will say ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes even if utf8mb4 is supported.

Did you allowed innodb large indexes ? For more details #2488180: Support full UTF-8 (emojis, Asian symbols, mathematical symbols) on MySQL and other database drivers when they are configured to allow it