API flaw - modified defaults don't affect existing records

Working on a test.

This patch does two things:

1) Excludes any tag from the flattening process if it matches the default of that field. This will exclude the tag from being serialized/saved.
2) Only falls back to default if the tag doesn't exist in the serialized string.

It does not address doing a mass updating of existing nodes.

I'm going to work on the script this morning. Figured I'd better put my name on it since there's a lot of cooks in this kitchen. ;)

After 4 hours of beating my head against this, I finally found a doc page that was my Rosetta Stone and gave me the start I needed to get this going. There is (obviously) more work to be done and I'll get to that tomorrow. This is the in-progress code:

// Get all of the field storage entities of type metatag.
$field_storage_configs = \Drupal::entityManager()->getStorage('field_storage_config')->loadByProperties(array('type' => 'metatag'));

foreach ($field_storage_configs as $key => $field_storage) {
	$field_name = $field_storage->getName();

    // Get the individual fields (field instances) associated with bundles.
	$fields = \Drupal::entityManager()->getStorage('field_config')->loadByProperties(array('field_name' => $field_name));
	
	foreach ($fields as $field) {
	  // Get the bundle this field is attached to.
	  $bundle = $field->getTargetBundle();

      // Get the default value for this field on this bundle.
      $field_default_tags_value = $field->getDefaultValueLiteral();
      $field_default_tags = unserialize($field_default_tags_value[0]['value']);

      // This part is only for nodes. Are we supporting other entities, yet?
      // Table name is node__FIELD_NAME
      // Field name with tags string is FIELD_NAME_value
      // Bundle is in field name bundle

      // Something like this to get rid of all the exact matches?
      // DELETE FROM {$node_field_table} WHERE bundle = {$bundle} AND {$value_field} = $field_default_tags_value
      
      // After that, need to prune on a field by field basis, comparing each individual tag

    }
}

Ok, I added an .install file to the patch that contains the update. I've tested it and it's working on the client data and leaving behind only the expected overridden values so I think it's good. But, it's one doozy of an update function including deleting and changing data in the table so it would be good to have more eyes on it. I especially don't know if this will cause any performance issues on sites with a huge amount of nodes. I couldn't think of a better way to make mass changes.

Ok, I'll work on that a bit later today. Need to spend some time on "C" before getting back to "M" again. :)

"small thing"... Ha... Brainbendy thing figuring out how to make nested foreach loops work with the sandboxing. Finally realized I could stuff the sandbox during the foreaches and then run the actual database code on the sandbox array. 3 hours later, I _almost_ have it working but it's only processing 834 of 839 records and then running out of records. So something is wrong somewhere. Going to post the code in case anyone else feels like having a look and I'll pick it up again tomorrow.

/**
 * Remove tags in field storage that match default or are empty.
 */
function metatag_update_8101(&$sandbox) {
  // This whole top section only needs to be done the first time.
  if (!isset($sandbox['records_processed'])) {
    $sandbox['records_processed'] = 0;
    $sandbox['total_records'] = 0;
    $sandbox['current_field'] = 0;
    $sandbox['current_record'] = 0;

    // Counter to enumerate the fields so we can access them in the array
    // by number rather than name.
    $field_counter = 0;

    // Get all of the field storage entities of type metatag.
    $field_storage_configs = \Drupal::entityManager()
      ->getStorage('field_storage_config')
      ->loadByProperties(array('type' => 'metatag'));

    foreach ($field_storage_configs as $key => $field_storage) {
      $field_name = $field_storage->getName();

      // Get the individual fields (field instances) associated with bundles.
      $fields = \Drupal::entityManager()
        ->getStorage('field_config')
        ->loadByProperties(array('field_name' => $field_name));

      // For each of the fields, do the mass delete of exact matches but
      // store the overridden records in the sandbox to be batch processed.
      foreach ($fields as $field) {
        // Get the bundle this field is attached to.
        $bundle = $field->getTargetBundle();

        // Get the default value for this field on this bundle.
        $field_default_tags_value = $field->getDefaultValueLiteral();
        $field_default_tags_value = $field_default_tags_value[0]['value'];
        $field_default_tags = unserialize($field_default_tags_value);

        // Determine the table and "value" field names.
        $field_table = "node__$field_name";
        $field_value_field = "$field_name" . "_value";

        // Delete all records where the field value and default are identical.
        db_delete($field_table)
          ->condition('bundle', $bundle, '=')
          ->condition($field_value_field, $field_default_tags_value, '=')
          ->execute();

        // Get all records where the field data does not match the default.
        $query = db_select($field_table);
        $query->addField($field_table, 'entity_id');
        $query->addField($field_table, 'revision_id');
        $query->addField($field_table, 'langcode');
        $query->addField($field_table, $field_value_field);
        $query->condition('bundle', $bundle, '=');
        $query->condition($field_value_field, $field_default_tags_value, '!=');
        $result = $query->execute();
        $records = $result->fetchAll();

        // Fill in all the sandbox information so we can batch the individual
        // record comparing and updating.
        $sandbox['fields'][$field_counter]['field_table'] = $field_table;
        $sandbox['fields'][$field_counter]['field_value_field'] = $field_value_field;
        $sandbox['fields'][$field_counter]['field_default_tags'] = $field_default_tags;
        $sandbox['fields'][$field_counter]['records'] = $records;

        $sandbox['total_records'] += count($sandbox['fields'][$field_counter]['records'] = $records);
        $field_counter++;
      }
    }
  }

  if ($sandbox['total_records'] == 0) {
    // No partially overridden fields so we can skip the whole batch process.
    $sandbox['#finished'] = 1;
  }
  else {
    // Begin the batch processing of individual field records.

    $max_per_batch = 10;
    $counter = 1;

    $current_field = $sandbox['current_field'];
    $current_field_records = $sandbox['fields'][$current_field]['records'];
    $current_record = $sandbox['current_record'];

    $field_table = $sandbox['fields'][$current_field]['field_table'];
    $field_value_field = $sandbox['fields'][$current_field]['field_value_field'];
    $field_default_tags = $sandbox['fields'][$current_field]['field_default_tags'];

    // Loop through the field(s) and remove any field data that matches the
    // field default for that bundle. Because the ability to override a default
    // with "nothing" didn't exist prior to this and because any tag that had
    // a default of "nothing" would have that also in the field data, we are
    // removing those as well.
    while ($counter <= $max_per_batch && $record = $current_field_records[$current_record]) {
      // Strip any empty tags or ones matching the field's defaults and leave
      // only the overridden tags in $new_tags.
      $current_tags = unserialize($record->$field_value_field);
      $new_tags = array();
      foreach ($current_tags as $key => $tag) {
        if (!empty($tag) && $field_default_tags[$key] != $tag) {
          $new_tags[$key] = $tag;
        }
      }

      if (empty($new_tags)) {
        // All tags were either empty or matched the default so the record can
        // be deleted.
        db_delete($field_table)
          ->condition('entity_id', $record->entity_id)
          ->condition('revision_id', $record->revision_id)
          ->condition('langcode', $record->langcode)
          ->execute();
      }
      else {
        // There are some overridden tags so update the record with just those.
        $tags_string = serialize($new_tags);
        db_update($field_table)
          ->fields(array(
            $field_value_field => $tags_string,
          ))
          ->condition('entity_id', $record->entity_id)
          ->condition('revision_id', $record->revision_id)
          ->condition('langcode', $record->langcode)
          ->execute();
      }

      $counter++;
      $current_record++;
    }

    if (!isset($current_field_records[$current_record])) {
      // We ran out of records for the field so start the next batch out with
      // the next field.
      $current_field++;
      $current_record = 0;
    }

    if (!isset($sandbox['fields'][$current_field])) {
      // We have finished all the fields. All done.
      $sandbox['#finished'] = 1;
    }
    else {
      // Update the sandbox values to prepare for the next round.
      $sandbox['current_field'] = $current_field;
      $sandbox['current_record'] = $current_record;
      $sandbox['records_processed'] += $counter-1;
      $sandbox['#finished'] = $sandbox['records_processed'] / $sandbox['total_records'];
    }
  }
}

Ok, I updated the update hook to use the sandbox. Let's give this patch a whirl.

+++ b/src/Plugin/Field/FieldWidget/MetatagFirehose.php
@@ -63,6 +63,10 @@ class MetatagFirehose extends WidgetBase {
+    // Get the field's default values.
+    $field_default_tags_value = $this->fieldDefinition->getDefaultValueLiteral();
+    $field_default_tags = unserialize($field_default_tags_value[0]['value']);
+
     // Flatten the values array to remove the groups and then serialize all the
     // metatags into one value for storage.
     foreach ($values as &$value) {
@@ -71,7 +75,11 @@ class MetatagFirehose extends WidgetBase {

@@ -71,7 +75,11 @@ class MetatagFirehose extends WidgetBase {
         // Exclude the "original delta" value.
         if (is_array($group)) {
           foreach ($group as $tag_id => $tag) {
-            $flattened_value[$tag_id] = $tag;
+            // Only include values that differ from the default.
+            // @TODO: When site defaults are added, account for those.
+            if ($tag != $field_default_tags[$tag_id]) {
+              $flattened_value[$tag_id] = $tag;

I don't this belongs in the widget. What if I build a new widget? I need to duplicate this logic. What if I submit via REST?.

I think this belongs in MetatagFieldItem::preSave().

Test for the bug part of this is in #2563637: Write tests for the 8.x-1.x functionality (commented out).

So once that's in, this issue should uncomment that assert.

Ok, I moved the default checking logic to the pre-save. I also discovered there's another problem with this UI of blank==overridden in that editing a node would override all the defaults since the defaults weren't automatically filled in on node edit. This patch makes sure they are all filled in when you edit a node. I still am not happy with that UI choice but I got it working.

I did not address the test as that hasn't yet been committed and I didn't want this dependent on another patch.

Missed a newline at the end of the .install file. See comment #25 for the rest on this patch.

Nice! the test passes for the overridden/default - but shows some warnings in the widget.

Working on fix.

tests++

Right so the fail is because the default value form in field ui doesn't yet have default values

tests++

working on detecting that and fixing

Fixes the default value issue.

Also removes the call to \Drupal in the constructor in favour of DI.

Do we have an issue for adding an interface for MetatagManager? Type-hinting a concrete implementation makes me feel dirty. If not, I'll add a new issue.

Added #2593141: Add a MetatagManagerInterface to avoid type-hinting concrete implementation