[PP-2] Field formatter & widget settings: fall back to default if missing plugin

Nice progress here 🙂

1. This will eventually need a change record for sure.
2. But right now, this really needs an issue summary update. The "proposed resolution" section explains at a high level what it aims to achieve, but doesn't explain how it achieves that in the current patch. The current patch is making significant changes, and as a non-expert in the migration system this is definitely not easy to understand. Explaining how you're changing the process pipeline and why you're changing it specifically this way would really help. (It'd help me to understand it and would probably make it easier for a migration system maintainer to review it too!) I'm definitely having trouble understanding the field_formatter_map_alter and field_widget_map_alter additions for example.

3. +++ b/core/modules/field/src/Plugin/migrate/process/FieldFormatterFallback.php
   @@ -0,0 +1,119 @@
   + * Transforms field formatter plugin id.
   ...
   +    // If the target field formatter plugin id is not available on the
   ...
   +      $message = sprintf('The field formatter plugin id %s (used on field type %s) could not be mapped to an existing formatter plugin; defaulting to %s and dropping all formatter settings. Either redo the migration with the module installed that provides an equivalent formatter plugin, or modify the entity view display after the migration and manually choose the right field formatter.', $value, $field_type, $fallback);
   
   +++ b/core/modules/field/src/Plugin/migrate/process/FieldWidgetFallback.php
   @@ -0,0 +1,107 @@
   + * Transforms field widget plugin id.
   ...
   +    // If the target field widget plugin id is not available on the
   

   🤓 s/id/ID/

4. +++ b/core/modules/field/src/Plugin/migrate/process/FieldFormatterFallback.php
   @@ -0,0 +1,119 @@
   + * Transform a non-mapped field formatter plugin_id to the D8 default formatter
   + * when the D8 target plugin_id is not available.
   ...
   +   *   The plugin_id for the plugin instance.
   
   +++ b/core/modules/field/src/Plugin/migrate/process/FieldWidgetFallback.php
   @@ -0,0 +1,107 @@
   + * Transform field widget plugin_id to the D8 default if the D8 target is not
   ...
   +   *   The plugin_id for the plugin instance.
   

   🤓
   s/plugin_id/plugin ID/
   s/target/destination/

5. +++ b/core/modules/field/src/Plugin/migrate/process/FieldFormatterFallback.php
   @@ -0,0 +1,119 @@
   +class FieldFormatterFallback extends ProcessPluginBase implements ContainerFactoryPluginInterface {
   ...
   +   * Constructs a FieldTypeDefaults plugin instance.
   

   🤓 Mismatch.

6. +++ b/core/modules/field/src/Plugin/migrate/process/FieldFormatterFallback.php
   @@ -0,0 +1,119 @@
   +   *   The formatter manager (no interface available).
   
   +++ b/core/modules/field/src/Plugin/migrate/process/FieldWidgetFallback.php
   @@ -0,0 +1,107 @@
   +   *   The field widget manager (no interface).
   

   You can omit the parenthetical here. It's fine that there is no interface and that you're hinting the concrete class :)

7. +++ b/core/modules/field/src/Plugin/migrate/process/FieldFormatterFallback.php
   @@ -0,0 +1,119 @@
   +    // The 'hidden' value is special in Drupal 7, and not a formatter.
   

   👍 Glad to see you're handling this edge case :)

   🤔 What about Drupal 6 though? I don't think this process plugin is D7-specific?

8. +++ b/core/modules/field/src/Plugin/migrate/process/FieldWidgetFallback.php
   @@ -0,0 +1,107 @@
   +class FieldWidgetFallback extends ProcessPluginBase implements ContainerFactoryPluginInterface {
   ...
   +   * Constructs a FieldWidgetDefaults plugin instance.
   

   🤓 Mismatch.

9. +++ b/core/modules/field/src/Plugin/migrate/process/FieldWidgetFallback.php
   @@ -0,0 +1,107 @@
   +      throw new MigrateException('Field type is required for the default field formatter.');
   ...
   +      throw new MigrateException('Field type is required for the default field formatter.');
   

   🐛 Widget, not formatter.

10. +++ b/core/modules/migrate/src/Plugin/migrate/process/NullIfNotEqual.php
    @@ -0,0 +1,51 @@
    + * Changes a source value to null if the two given variable are not equal.
    

    🤓 s/variable/variables/

#12:

1. Yes, of course that's not necessary yet. Just wanted to add the tag so we don't forget :)
2. 👍 Super helpful, thank you! But I'm still not clear on the new "alter" entries in the migration definition YAML files.

7. 👍 We'll let a migration system maintainer sign off on this. Tagging for that.

(This is relatively minor, let's focus on everything else first.)

These are caught in MigrateUpgradeImportBatch and logged into the logger.

But that means they're logged on behalf of migrate_drupal_ui instead of on behalf of the specific migration plugin where it occurred.

Plus, it means that it'd be logged differently for each migration runner, for example https://www.drupal.org/project/migrate_tools or https://www.drupal.org/project/migrate_run would yield different log entries.

1. +++ b/core/modules/field/migrations/d6_field_formatter_settings.yml
   @@ -276,7 +285,6 @@ process:
   -
   
   +++ b/core/modules/field/migrations/d6_field_instance.yml
   @@ -46,7 +46,6 @@ process:
   -
   
   +++ b/core/modules/field/migrations/d6_field_instance_widget_settings.yml
   @@ -60,14 +61,28 @@ process:
   -
   

   🤓 Übernit: removing empty lines in this patch is an unnecessary distraction in an already complex patch. Let's not change these.

2. +++ b/core/modules/field/migrations/d7_field_instance.yml
   @@ -38,18 +38,45 @@ process:
   +    # @TODO: add the same mapping as d7_field_instance_widget_settings has.
   

   Looks like this still needs to happen.

3. +++ b/core/modules/field/migrations/d7_field_instance.yml
   @@ -38,18 +38,45 @@ process:
   +      plugin: field_widget_fallback
   ...
   +      skip_message: true
   
   +++ b/core/modules/field/migrations/d7_field_instance_widget_settings.yml
   @@ -61,11 +62,27 @@ process:
   +      plugin: field_widget_fallback
   

   🤔 Why do we skip the message only sometimes?

4. +++ b/core/modules/field/migrations/d7_field_instance.yml
   @@ -38,18 +38,45 @@ process:
   +    -
   +      plugin: null_if_not_equal
   +      values_to_compare:
   +        - '@widget_with_fallback'
   +        - 'widget/type'
   +    -
   +      plugin: default_value
   +      default_value: []
   
   +++ b/core/modules/migrate/src/Plugin/migrate/process/NullIfNotEqual.php
   @@ -0,0 +1,49 @@
   +class NullIfNotEqual extends ProcessPluginBase {
   

   👏 This looks elegant to me :) But definitely needs migration system maintainer sign-off.

5. +++ b/core/modules/field/src/Plugin/migrate/process/FieldWidgetFallback.php
   @@ -0,0 +1,125 @@
   + * widget plugin id is unavailable.
   ...
   +        $message = sprintf('The field widget plugin id %s (used on field type %s) could not be mapped to an existing widget plugin; defaulting to %s and dropping all widget settings. Either redo the migration with the module installed that provides an equivalent plugin, or modify the entity form display after the migration and manually choose the right field widget.', $value, $field_type, $fallback);
   

   🤓 Übernit: s/id/ID/

Manually tested this patch on two test cases: wimleers.com (simple D7 site) and Demo Framework (complex D7 site).

wimleers.com

Before

After

Demo Framework

Before

After

Conclusion

1. Big improvement in Demo Framework! 🥳 But … some still remain. Presumably because they're for custom field types? (We of course cannot compute a default field formatter for a field type we don't know!)
   I think that in this case we should detect that it's for an unknown field type and provide an error message attuned to that. That would prevent users from wasting time on this if the real problem to solve is installing the appropriate field type — because if no fallback can be computed, then the corresponding FieldConfig and FieldStorageConfig surely also failed!
2. No changes for wimleers.com. This seems wrong … because at least some of them are for taxonomy reference fields, which can be migrated.

Both of these need to be improved, but keeping at Needs review because @huzooka's question in #22 still stands: this needs sign-off from a migration system maintainer for the general direction.

Came up in yesterday's Migrate meeting, no response yet. We'll keep waiting patiently :)

Manually tested with wimleers.com again.

With #23

With #26

And this positive impression is confirmed if we dig a little deeper:

i.e. all of the missing formatters now have a message saying we automatically fell back to the default! 🥳

First, four questions, "just checking":

1. Is #23.1 handled?
2. Should the code comments in alterFieldWidgetMigration() and alterFieldFormatterMigration() "match" (so to speak)? -- right now, the former has more extensive comments than the latter, but the code the comments describe seems to be the same.
3. Sorry if I missed it in the thread -- has anyone tried using null_if_not_equal in another situation other than this fallback-field-formatter/widget-plugins use case?
4. I'm sure it would've said somewhere prominent if there were "breaking changes," but is there a brief explanation y'all might be able to offer for why the following two diffs won't break anything...? (asking b/c I'm having a go at a change record, to help move the issue along)
   These two diffs, near line 82 (alterFieldWidgetMigration) -- and they're repeated for the "formatter" function, of course:
   1:

   -      $process['type']['map'][$source_widget] = $destination_widget;
   +      $process['map'][$source_widget] = $destination_widget;
   

   2:
   - $migration->mergeProcessOfProperty('options/type', $process);

Second, I have a draft change record -- but I'm going to post it in a follow-up comment, b/c it got pretty long and I don't want things to get lost. I'm sure it can be trimmed down, this is just a first draft.

"Draft change record", such as it is... coming at this issue fresh, doing my best to understand what's being done! (FWIW, I'm least sure of my explanation of null_if_not_equal.)

Summary

Use default field formatter plugins and field widget plugins when source formatters/widgets don't have a replacement on the destination site, thereby preventing affected fields from being hidden in its target entity_*_display (view or form), and making these fields usable and visible.

Also, the new null_if_not_equal process plugin can be used outside this widget/formatter fallback use case.

Before

When Drupal migrates field formatter and field widget configurations into entity_view_display and entity_form_display entities, it often happens that the migrated formatter plugin (or field widget) has no replacement on the destination site. In these cases, plugin discovery throws (and logs) a PluginNotFoundException (with an another log created by MigrateUpgradeImportBatch::onIdMapMessage) message (MESSAGE_ERROR severity).

Also, the affected field gets hidden in its target entity_view_display or entity_form_display entity.

After

Automatically select the default field formatter plugin ID or widget plugin ID. (A migrate message is still saved to logs, saying that the "fallback" was used, but just one message, and with MESSAGE_NOTICE severity.)

More details

Three new migrate process plugins have been added:

1. FieldFormatterFallback: Transform a non-mapped field formatter plugin ID to the D8 default formatter when the destination plugin ID is not available.
2. FieldWidgetFallback: Transforms field widget plugin ID to the D8 default widget if the destination widget plugin id is unavailable.
3. NullIfNotEqual: Changes the source value to null if the two given variables are not equal.

(The first two, the "fallback" plugins, save a migrate message with MigrationInterface::MESSAGE_NOTICE severity on fallback.)

The third plugin, null_if_not_equal, was added in order to compare the original (mapped-only) "widget/formatter_type_mapped" value with the final/"real" value ('options/type'), and return an empty array. The empty array prevents the widget/formatter plugin from being migrated -- instead, the fallback/default plugin values are used. In other words, if the compared values don't match, the plugin settings are set to "null", which triggers the fallbacks.

The null_if_not_equal process plugin can be used beyond this widget/formatter fallback use case.

Lastly, two new static_map process plugin options were added to FieldPluginBase -- both boolean -- to allow merging the widget/formatter mapping for use by the new fallbacks:
field_widget_map_alter
field_formatter_map_alter

Just a drive-by review here.

NullIfNotEqual is probably not needed. I haven't worked out how, but this type of thing is usually possible without the addition of new plugins.

Instead of the existing somewhat complicated method to merge things, couldn't the existing static_map class be extended by FieldFormatterFallback and FieldWidgetFallback and build all that logic internally?

For the CR, usually we use the name of the plugin_id, as that is part of the API and the class name isn't so important.

(Thanks for the CR feedback! I'll integrate that into the actual draft CR if this whole thing moves forward.) (Meanwhile...)

...........
I realize #3061571: If no Drupal 8 equivalent filter is found during migration, drop transformation-only filters, so that body fields show up is different, but it just got bumped to RTBC -- since we're getting maintainer concerns on this issue (#3108302), I just wondered, even though they're different situations, are there any "principles" from the implementation on #3061571 that could/should be applied to this issue (#3108302)...?

Ignore #30 now. I've spent a little more time looking into things.

1. +++ b/core/modules/migrate/src/Plugin/migrate/process/NullIfNotEqual.php
   @@ -0,0 +1,49 @@
   + *
   + * @MigrateProcessPlugin(
   + *   id = "null_if_not_equal"
   + * )
   + */
   +class NullIfNotEqual extends ProcessPluginBase {
   

   So, I did some poking around in what we currently have in core for comparison process plugins. There doesn't seem to be any. And using callback, it only supports a single value, not multiple arguments. Skip on value in migrate_plus also doesn't fit the bill either.

   So I got to thinking, what if instead of making a single purpose plugin, what if we created a "comparison" process plugin? One that had as config the traditional compare operators: https://www.php.net/manual/en/language.operators.comparison.php and returned true/false. From there, we could easily do a static_map mapping of true/false to empty array.

   Remember that transform plugins can take more then one value into the transform method. See file_copy as an example.

2. And for FieldFormatterFallback/FieldWidgetFallback, could the logic in there be

   +++ b/core/modules/field/migrations/d6_field_instance_widget_settings.yml
   @@ -40,32 +40,44 @@ process:
   +    plugin: static_map
   +    field_widget_map_alter: true
   
   +++ b/core/modules/field/migrations/d7_field_instance_widget_settings.yml
   @@ -44,28 +44,41 @@ process:
   +    plugin: static_map
   +    field_widget_map_alter: true
   
   +++ b/core/modules/migrate_drupal/src/Plugin/migrate/field/FieldPluginBase.php
   @@ -82,9 +82,28 @@ public function processFieldWidget(MigrationInterface $migration) {
   +        if (empty($process_plugin->configuration['field_widget_map_alter']) || $process_plugin->getPluginId() !== 'static_map') {
   +          continue;
   +        }
   +        // If this is a static map tagged with 'field_widget_map_alter', we
   +        // merge the previously discovererd widget mapping.
   +        // We get normalized process plugins. If the source is set in this
   +        // static_map plugin, we have an (implicit) preceding get plugin, and
   +        // we have to lower the destination plugin key by 1.
   +        $key = isset($process_plugin->configuration['source']) && is_numeric($process_plugin_key) ?
   +          $process_plugin_key - 1 : $process_plugin_key;
   +        $migration->mergeProcessOfProperty($destination_key, [$key => $process]);
   
   +++ b/core/modules/migrate_drupal/tests/src/Kernel/d6/FieldDiscoveryTest.php
   @@ -180,12 +182,23 @@ public function addAllFieldProcessesAltersData() {
   +              'plugin' => 'static_map',
   +              'field_widget_map_alter' => TRUE,
   
   @@ -200,6 +213,15 @@ public function addAllFieldProcessesAltersData() {
   +              'plugin' => 'static_map',
   +              'field_widget_map_alter' => TRUE,
   
   +++ b/core/modules/migrate_drupal/tests/src/Kernel/d7/FieldDiscoveryTest.php
   @@ -241,12 +244,23 @@ public function addAllFieldProcessesAltersData() {
   +              'plugin' => 'static_map',
   +              'field_widget_map_alter' => TRUE,
   
   @@ -265,6 +279,15 @@ public function addAllFieldProcessesAltersData() {
   +              'plugin' => 'static_map',
   +              'field_widget_map_alter' => TRUE,
   

   I'm not sure why we have to add the boolean field_widget_map_alter flag. Could we leverage the existing call to getFieldWidgetMap and add the mappings into it?

#35: wow, this patch now looks way simpler! Well done :)

1. +++ b/core/modules/field/src/Plugin/migrate/process/FieldPluginFallback.php
   @@ -0,0 +1,172 @@
   +class FieldPluginFallback extends ProcessPluginBase implements ContainerFactoryPluginInterface {
   

   👍 This is the generic successor to both FieldFormatterFallback and FieldWidgetFallback in #26.

2. +++ b/core/modules/field/src/Plugin/migrate/process/FieldPluginFallback.php
   @@ -0,0 +1,172 @@
   + * Transforms field formatter plugin ID.
   + *
   + * Transform a non-mapped field formatter plugin ID to the D8 default formatter
   + * when the destination plugin ID is not available.
   + *
   

   🐛 Needs doc update: this works not just for field formatter plugin IDs now, but also field widget plugin IDs.

   👉Fixed in attached patch.

3. +++ b/core/modules/field/src/Plugin/migrate/process/FieldPluginFallback.php
   @@ -0,0 +1,172 @@
   +        case 'field_widget':
   +          if (!$this->widgetPluginManager->hasDefinition($value)) {
   +            $message = sprintf(
   +              'The field formatter plugin ID %s (used on field type %s) could not be mapped to an existing formatter plugin; defaulting to %s and dropping all formatter settings. Either redo the migration with the module installed that provides an equivalent formatter plugin, or modify the entity view display after the migration and manually choose the right field formatter.',
   +              $value,
   +              $field_type,
   +              $field_type_definition['default_widget']
   

   🐛 This message is wrong: this is for widgets, not formatters.

   👉Fixed in attached patch.

4. +++ b/core/modules/field/src/Plugin/migrate/process/FieldPluginFallback.php
   @@ -0,0 +1,172 @@
   +    $plugin_type = $configuration['field_plugin'] ?? NULL;
   +    if (
   +      empty($plugin_type) ||
   +      !in_array($plugin_type, ['field_formatter', 'field_widget'], TRUE)
   +    ) {
   +      throw new MigrateException("Field plugin type is a required configuration of the 'field_plugin_fallback' migrate process plugin.");
   +    }
   

   👍

I did not mean to change the IS.

While I'm here, I might as well clarify the issue title: it is blocked on #3202462: [PP-1] Provide option for contrib modules to map their D6 / D7 field formatter and widget plugin IDs to the equivalent D9 plugin ID, which is in turn blocked on #3204212: Convert remaining widget and formatter type migrations to MigrateField plugins .

Confirming #40 fixed my migration CCK file (image) field problems from Drupal 6 to Drupal 9.1.7 where I used "Lightbox 2" as image formatter, which wasn't existing in Drupal 9. Without the patch the Gallery images were missing, now they are imported with the fallback as expected.

For others who need the whole composer patches combo of this issue with current state:

"patches": {
  "drupal/core": {
    "[Migration:] Move hard-coded field widget ID and formatter ID mapping from d*_field_instance_widget_settings and d6_field_formatter_settings migrations to the corresponding MigrateField plugins": "https://www.drupal.org/files/issues/2021-06-14/3204212-field-migration-widget-formatter-mapping-27.patch",
    "[Migration:] [PP-1] Provide option for contrib modules to map their D6 / D7 field formatter and widget plugin IDs to the equivalent D9 plugin ID": "https://www.drupal.org/files/issues/2021-04-08/core-allow_map_formatter_migration-3202462-8--on-top-of-3204212.patch",
    "[Migration:] [PP-2] Field formatter & widget settings: fall back to default if missing plugin": "https://www.drupal.org/files/issues/2021-04-15/core-migrate_field_formatter_widget_with_fallback-3108302-40--on-top-of-3202462-8_0.patch"
  }
}

BTW shouldn't this be "Needs review", not "Needs work" as of #40? (Or alternatively "Postponed"?)

EDIT: Updated to work with 9.3.x, replaced https://www.drupal.org/files/issues/2021-03-22/3204212-field-migration-w...

Rebased #40 to apply to 9.2.0-rc1.

Was trivial thanks to git :)

#44: yay!!! 🥳

Hopefully we can get this committed soon!

Yay!

We still need a CR, right? (It's not just an outdated tag?)

If yes, if no one else has already started, I can work on updating the one I started in #29, with the feedback in #30 -- but could someone offer some guidance on "where" I should "put" a draft CR? -- just in a comment, or in the issue summary, or an actual CR node, or?

@alisonjo315, The actual CR. I use the link at the top of the Issue just above 'Related issues', 'add change notice', the change notice will open in draft mode.

Anybody knows why the patch from #45 fails to apply on 9.3.x? Just wanted to use the patch for my local d6 Migration, but can't seem to be able to apply it. Is it dependent on some other patch? If so which one?

I tried the patch-order from @Anybody's comment in #44, but then it doesn't want to apply the patch from #3204212: Convert remaining widget and formatter type migrations to MigrateField plugins . Maybe somebody can help? I'm currently using Drupal 9.3.0-alpha1.

Sorry if this is trivial stuff :)

@Grevil: #44 contained the first patch against 9.1.x - updated and replaced by
https://www.drupal.org/files/issues/2021-06-14/3204212-field-migration-w...

Please try if all three apply now or which one doesn't

@Anybody still can't apply patch "https://www.drupal.org/files/issues/2021-06-14/3204212-field-migration-w..." from #3204212: Convert remaining widget and formatter type migrations to MigrateField plugins , even with rerolled patch.

Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2021-06-14/3204212-field-migration-widget-formatter-mapping-27.patch

EDIT: Apparently patch 27 from #3204212: Convert remaining widget and formatter type migrations to MigrateField plugins doesn't work for 9.3.x anymore, see comment #27. It still works for 9.2.x though!

Rerolled patch #26 against drupal core 9.5.x

Rerolled patch #26 with Drupal 9.5.x

It could be useful for #3213491-20: Add fallback format support to responsive images

This one definitely needs a status change. All the recent patches have failures, so at the very least this should be set to Needs Work. But this also appears to be blocked by two other issues per #41. If this isn't in fact postponed, it needs a title update. I'm going to be bold and set this to Postponed.

This patch (I'm still using #45, and that works fine even on Drupal 10.2!) triggers

Creation of dynamic property Drupal\field\Plugin\migrate\process\FieldPluginFallback::$migration is deprecated

on PHP >=8.2.

Fixed that.

The Migrate Drupal Module was approved for removal in #3371229: [Policy] Migrate Drupal and Migrate Drupal UI after Drupal 7 EOL.

This is Postponed. The status is set according to two policies. The Remove a core extension and move it to a contributed project and the Extensions approved for removal policies.

The deprecation work is in #3522602: [meta] Tasks to remove Migrate Drupal module and the removal work in #3522602: [meta] Tasks to remove Migrate Drupal module.

Migrate Drupal will not be moved to a contributed project. It will be removed from core after the Drupal 12.x branch is open.

The Migrate Drupal and Migrate Drupal UI modules are deprecated and will be removed from Drupal 12. For these modules, effort is now focused on bug fixes and necessary tasks. Therefore, this feature request is closed as won't fix.

Thanks to all for working on this!