Respect original default value, not null and comment definitions

Hmm, that's no good -- this will need a patch, and ideally an upgrade path for anyone else with the same problem.

For now, something like this might do (very ugly though).

We'll also have to provide an upgrade path for people who have already run the script, and who don't have the option to recover from backups and run a newer version of the script. I guess we'll have to use the schema.

Problem is, they already might have inconsistent data because of other defaults/not null options :(

Emergency beta2 release which includes this fix: https://www.drupal.org/project/utf8mb4_convert/releases/7.x-1.0-beta2

It's very unfortunate this took 6 weeks to be found, and 2000 people downloaded "broken" versions already.

If at all possible, for now (while we don't have any upgrade path) the recommended way to go is to recover old backups, and re-run the script using beta2 or later.

Back to Critical. Included a mention of this issue on the project page as well.

WARNING: obviously(!), don't run this without backups or testing

I ran this with the Schema module installed to fix my field definitions:

// Get the schema
$schema = drupal_get_schema(NULL, TRUE);

// Get the database schema
$db_schema = schema_dbobject()->inspect();

// Loop through each declared schema table
foreach ($schema as $table_name => $schema_table) {
  $db_table = $db_schema[$table_name];

  // Loop through each column
  foreach ($schema_table['fields'] as $colname => $col) {
    $db_col = $db_table['fields'][$colname];

    // Only touch varchar, char, and text
    if (!in_array($col['type'], array('varchar', 'char', 'text'))) continue;

    if (
      // Not null is/isn't set
      (isset($col['not null']) != isset($db_col['not null'])) ||

      // Not null set in schema and set in db; doesn't match
      (isset($col['not null']) && isset($db_col['not null']) && $col['not null'] != $db_col['not null']) ||

      // Default is/isn't set
      (isset($col['default']) != isset($db_col['default'])) ||

      // Default set in schema and set in db; doesn't match
      (isset($col['default']) && isset($db_col['default']) && $col['default'] != $db_col['default']) ) {

      // Fix nulls if set to not null and there's a default.
      // This prevents errors updating tables with new information that may have nulls where they aren't allowed.
      if (isset($col['not null']) && $col['not null'] && isset($col['default'])) {
        db_update($table_name)
          ->fields(array(
            $colname => $col['default'],
          ))
          ->isNull($colname)
          ->execute();
      }

      // Update field in db
      db_change_field($table_name, $colname, $colname, $col);
    }
  }
}

The code in #9 does seem to restore the correct NOT NULL and DEFAULT settings, but not the descriptions.

According to Jane_wu's comments on the drupal_get_schema() documentation, it doesn't actually return any descriptions:

Curious on why not returning the column description with other information at the same time?
In my cases, I'd like to update the column description. Now I have to copy and paste the description T_T

It seems in my cases, I'd better to use drupal_get_schema_unprocessed to get exactly what's come up from hook_schema and hook_schema_alter

Further to my previous comment, I replaced the call to drupal_get_schema() with some relevant code ripped from drupal_get_complete_schema() and then added FALSE to the _drupal_schema_initialize() call so that it doesn't remove the descriptions.

This seems to work better:

// Initialise schema.
$schema = array();

// Include all module install files.
module_load_all_includes('install');

// Get complete schema.
foreach (module_implements('schema') as $module) {
  $current = (array) module_invoke($module, 'schema');
  _drupal_schema_initialize($current, $module, FALSE);
  $schema = array_merge($schema, $current);
}

// Apply schema modifications.
drupal_alter('schema', $schema);

// Get the database schema
$db_schema = schema_dbobject()->inspect();

// Loop through each declared schema table
foreach ($schema as $table_name => $schema_table) {
  $db_table = $db_schema[$table_name];

  // Loop through each column
  foreach ($schema_table['fields'] as $colname => $col) {
    $db_col = $db_table['fields'][$colname];

    // Only touch varchar, char, and text
    if (!in_array($col['type'], array('varchar', 'char', 'text'))) continue;

    if (
      // Not null is/isn't set
      (isset($col['not null']) != isset($db_col['not null'])) ||

      // Not null set in schema and set in db; doesn't match
      (isset($col['not null']) && isset($db_col['not null']) && $col['not null'] != $db_col['not null']) ||

      // Default is/isn't set
      (isset($col['default']) != isset($db_col['default'])) ||

      // Default set in schema and set in db; doesn't match
      (isset($col['default']) && isset($db_col['default']) && $col['default'] != $db_col['default']) ||
      
      // Description is/isn't set
      (isset($col['description']) != isset($db_col['description'])) ||
      
      // Description set in schema and set in db; doesn't match
      (isset($col['description']) && isset($db_col['description']) && $col['description'] != $db_col['description']) ) {

      // Fix nulls if set to not null and there's a default.
      // This prevents errors updating tables with new information that may have nulls where they aren't allowed.
      if (isset($col['not null']) && $col['not null'] && isset($col['default'])) {
        db_update($table_name)
          ->fields(array(
            $colname => $col['default'],
          ))
          ->isNull($colname)
          ->execute();
      }

      // Update field in db
      db_change_field($table_name, $colname, $colname, $col);
    }
  }
}

Can we turn this into a patch so that we can run this by doing "drush utf8mb4-convert-fix"?

I'm not familiar with drush authoring but I'll have a look at the documentation on drush.org and see if I can provide a patch.

To turn this into a drush command, do we need to get rid of the dependency on schema_dbobject()->inspect() from schema.module?

I'm OK with keeping the dependency -- just fail with a drush_print() telling people to install the schema module if it's not installed.

Ok, here's my first attempt at a patch to add the new command - I hope it's suitable.

Would it have been preferable to add another function to the DrupalCharsetConverter class and use $this->connection->query() syntax?

Anyway, I've only tested it on a single server running a single site, but it seems to do the trick.

Re-roll of previous patch without email headers

Should this be closed if it's been committed?