Process plugin: snippet to re-use YAML config

From the project description:

MigrationGroup configuration entities [...] allow configuration to be shared among multiple migrations.

Now, admittedly, this is limited, since migrations cannot belong to multiple groups, AFAICT. But it does allow you to significantly cut down on duplicate config.

Would any of the guidance for re-use of yaml found in https://medium.com/@kinghuang/docker-compose-anchors-aliases-extensions-... be helpful? It doesn't let you re-share across yamls, but it does let you share within the same yaml file.

I've often thought it would be great if YAML had a preprocessor like C, with #include and #define. But, of course, the C preprocessor can be used with any kind of file - so why reinvent the wheel? Now, reading that, perhaps m4 (which I've never used) would be a better choice.

The point being - there's no need to implement something custom here, just someone to write up a tutorial on using cpp or m4 with YAML. Maybe develop some conventions and best practices for migration files, a repository of snippets for common use cases, etc.

Hi, I have test the code and it works like a charm! It's being very useful for me and I think that is a very simple tool that can be shared to the community.

I'm not sure, @mykeryan, if we can resolve this requirement using a C preprocessor, cpp or something like that. Really, I don't know. Maybe we can do it, but I think that developers prefer to use a simple process plugin like this.

BTW, Herve, thank for sharing! My two cents is that incorporate this to migrate_plus is a good idea! :)

I do like not reinventing the wheel so I tested a bit with using m4 as suggested by mikeryan in #7. I created a small migration yml, test.yml, and an m4 file, sample.m4, containing a definition of a pipeline with two destination fields. In that file the destination fields names are set as $1 and $2 so they can be filled in when the command run. Just thought it would be nice to see some flexibility here.

test.yml

id: sample
source:
  plugin: sample
process:
  id: sample_id
  pipeline_sample(dest1, dest2)
destination:
  plugin: entity:sample

sample.m4

define(`pipeline_sample',
  $1:
    -
      plugin: callback
      source:
        - name_translated
        - name
      callable: array_filter
    -
      plugin: callback
      callable: current
  $2:
    -
      plugin: callback
      source:
        - description_translated
        - description
      callable: array_filter
    -
      plugin: callback
      callable: current
)

Now process test.yml with the sample.m4

v@dev3 {/opt/sites/d8/sample} (m4)$ m4 sample.m4 test.yml > out.yml
v@dev3 {/opt/sites/d8/sample} (m4)$ cat out.yml

id: sample
source:
  plugin: sample
process:
  id: sample_id
  dest1:
    -
      plugin: callback
      source:
        - name_translated
        - name
      callable: array_filter
    -
      plugin: callback
      callable: current
  dest2:
    -
      plugin: callback
      source:
        - description_translated
        - description
      callable: array_filter
    -
      plugin: callback
      callable: current

destination:
  plugin: entity:sample

    -
      plugin: callback
      source:
        - description_translated
        - description
      callable: array_filter
    -
      plugin: callback
      callable: current

destination:
  plugin: entity:sample

That is certainly easy. Is this worth pursuing?

The great gitlab migration is upon us! This is now a duplicate of https://gitlab.com/drupalspoons/migrate_plus/-/issues/133.

Now that this issue queue is active again, I think we should re-open this issue.

We have a choice of approaches to take!

New configuration type

Use the pipeline process plugin from the patch in #2 and pipelines defined in migrate_plus.migration_process_pipeline.*.yml.

I think it will work only with migrations defined as config entities (using the Migrate Plus module), not core migrations (defined using mymodule/migrations/*.yml). Maybe not. It will certainly require the Migrate Plus module.

YAML anchors

See #5, #6.

I think this only works to reuse a pipeline within a single YAML file.

Do we have any working examples? I am not sure that the YAML parser we currently use (Symfony?) supports anchors.

Preprocessor

See #7, #10.

This requires some sort of build process or manually running the preprocessor.

New process plugin and migrations/process/*.yml

The snippet plugin from #3214186: Scan for snippets of process plugins is similar to the pipeline plugin, but it looks for "snippets" in mymodule/migrations/process/*.yml instead of in config entities.

This is currently a patch on the Migrate Scanner module, but it could easily be moved here. It will work like any other process plugin. I may be biased, but I think the implementation is simpler than for the pipeline plugin.

Using a specific directory may be unfamiliar for people used to working with Migrate Plus configuration.

New key in the migration YAML

@mikelutz proposed this in #3214258: [meeting] Migrate Meeting 2021-05-20.

The idea is to allow a new key, process_pipelines, in the migration YAML. Presumably, this would work with either core migrations or migrations defined by config entities.

Example:

id: my_migration
source:
  plugin: sql
process:
  name:
    pipeline: trim_and_capitalize
	source: name
  title:
    pipeline: trim_and_capitalize
	source: title
destination:
  plugin: entity:node
process_pipelines:
  trim_and_capitalize:
    - 
	  plugin: callback
	  callable: trim
	-
	  plugin: callback
	  callable: ucwords

Using just Drupal core, a pipeline could be reused within a single migration, as in the example.

Using Migrate Plus, the pipelines could be defined in a migration group and shared with multiple migrations.

I think this would have to be implemented as a core patch. Is there any use for it in core? A good place to look is the List of core Migrate process plugins, especially the ones outside the Migrate module.

I think this would be more complicated to implement than the pipeline or snippet plugin.

Now that this issue queue is active again

What did I miss? I know these issues were on drupalspoons before. Is there any announcement of the issues moving back here?

I think it will work only with migrations defined as config entities

In the project we are currently working on, we are using patch #2 successfully, with migrations as config entities.
We maintain the migrations in a module config/install dir, and run a selective drush config-import to update them.

However, I think all that we are doing there could have also been achieved with migrations declared as plugins with yml. I don't really know what we gained by using config entities.

So perhaps a pipelines solution should also support a setup where migrations are not config entities.

YAML anchors
See #5, #6.
I think this only works to reuse a pipeline within a single YAML file.

In most of the cases where we use pipelines, we reuse them across yml files. Being limited to the same yml file would defeat the purpose in our case.

Preprocessor

I think I prefer processing plugin definitions rather than mess with the yml language.

New process plugin and migrations/process/*.yml
The snippet plugin from #3214186: Scan for snippets of process plugins is similar to the pipeline plugin, but it looks for "snippets" in mymodule/migrations/process/*.yml instead of in config entities.

Not sure what exactly is being proposed here, but:
We could use a plugin deriver to register "composite processors" as new process plugins. This would achieve the same goal as the pipeline. There could be an option to make this a config entity.

New key in the migration YAML
Using Migrate Plus, the pipelines could be defined in a migration group and shared with multiple migrations.

This would restrict reuse of the pipeline to the migrations within the same group. This seems unnecessarily restrictive to me, and would also reduce flexibility for how we can use groups.
I would prefer if this was not tied to migration groups.
In fact I would prefer some kind of inheritance over migration groups shared_configuration.

@donquixote:

Thanks for the input!

I know these issues were on drupalspoons before. Is there any announcement of the issues moving back here?

It was announced on the #migration channel in Drupal Slack. Is there another place where you would be more likely to notice the announcement? Was the move to DrupalSpoons announced somewhere?

I would like to know for sure whether the pipeline plugin from #2 works with core migrations. Can you do a test?

Sample usage for the snippet plugin (simplified version of the example in #3214186: Scan for snippets of process plugins):

mymodule/migrations/process/clean_whitespace.yml:

-
  plugin: callback
  callable: htmlentities
-
  plugin: str_replace
  search:
    - ' '
    - ' '
  replace: ' '

That is used in this migration, mymodule/migrations/test.yml:

process:
  name:
    - plugin: snippet
      module: mymodule
      path: clean_whitespace
      source: string

To be clear: "I may be biased" means "I wrote this."

I considered a plugin deriver, but I decided that it was simpler to have a single plugin and read migrations/process/*.yml from that plugin.

It was announced on the #migration channel in Drupal Slack. Is there another place where you would be more likely to notice the announcement? Was the move to DrupalSpoons announced somewhere?

I don't know what would be the "correct" place to do this.
One possibility would be to have an issue either here or on drupalspoons, and then a short note on the module page with a link to the issue.
Could also be a wiki page or a change record. I don't know if any of this is the ideal solution, but at least issues are easy to create.

I would like to know for sure whether the pipeline plugin from #2 works with core migrations. Can you do a test?

I never used the core migrations, so I would not even know where to start :)
Also, do we need pipelines to work with core migrations?

I considered a plugin deriver, but I decided that it was simpler to have a single plugin and read migrations/process/*.yml from that plugin.

I don't have a clear opinion whether I prefer a plugin deriver or a single plugin with a setting.
However, if the files are like "$module_dir/migrations/process/*.yml", then I would expect these to be process plugins.
If they are just options within the snippet plugin, I would name them e.g. "$module_dir/migrations/snippets/*.yml".

Seems related to #3250040: Shared configuration for migrations.

I would like to know for sure whether the pipeline plugin from #2 works with core migrations. Can you do a test?

I did a quick test of patch #2 using Migrate Sandbox (which doesn't rely on migrate_plus to run migrations) and it worked fine.

My "Migration Process Pipeline" plugin (migrate_plus.migrate_process_pipeline.my_process_pipeline.yml) ended up looking like this:

uuid: cad0d398-5a11-4e98-8d88-98eb88f83b41
langcode: en
status: true
dependencies: {  }
id: my_process_pipeline
label: 'My process pipeline'
description: 'Process pipeline for processing fulltext fields'
module: null
process:
  -
    plugin: callback
    callable: strtoupper
  -
    plugin: explode
    delimiter: X

It obviously doesn't do anything useful, but it's fine for a test!

My process pipeline looked like this:

testing:
  plugin: pipeline
  id: my_process_pipeline
  source: data

I also used the patch for migrate_scanner (though it no longer applies to HEAD) and had similar success. I put the following in my_module/migrations/process:

-
  plugin: callback
  callable: strtoupper
-
  plugin: explode
  delimiter: X

The process pipeline read like so:

testing:
  plugin: snippet
  module: my_module
  path: my_process_pipeline
  source: data

These two approaches don't differ that much from from the outside at least. I guess I think the migrate_scanner approach is a little bit simpler for me. It's clumsy to get new config entities discovered whereas the snippets get discovered easily.

I personally find #2 and the migrate_scanner approach easier to grok than the approaches in #10. But I also understand the sentiment of not wanting to reinvent the wheel.

Here's the migrate_plus patch version of @benjifisher's MR against migrate_scanner, creating the Snippet process plugin described in #12 and #14. This is 100% the work of @benjifisher.

I ran into a little bug using the snippet process plugin in the patch in #18, specifically in the case where a process plugin needs the migration passed to it (like migration_lookup). This fixes that problem.

This doesn't work with sub_process plugin and/or when sending in an array. The snippet plugin ends up stripping the 1st item out of the source array from the main migration yml rather than the whole array, so the snippet file only gets the 1st item.

Would it work to put the single_value plugin in the pipeline ahead of the snippet plugin in that case?

testing:
  -
    plugin: single_value
    source: my_array
  -
    plugin: snippet
    module: my_module
    path: my_process_pipeline

@danflanagan8 that seems to be working! What a nice workaround. Thank you.

Snippet YML:

-
  plugin: sub_process
  source: bricks
  process:
    _stub:
      plugin: switch_on_condition
      source: type
      .....

--migrate-debug output:

...
    13 => []
    14 => array:3 [
      "_stub" => array:2 [
        0 => "227"
        1 => "274"
      ]
      "target_id" => "227"
      "target_revision_id" => "274"
    ]

It's interesting how the output values contain the pseudo_field, which usually isn't present in the output for Destinations. It has no effect on the import thankfully.

I just started using #19 to do identical processing on a bunch of rich text fields and I'm in love, this feature is absolutely wonderful, THANK YOU!!

The status is "Needs work" not "Needs review," so I won't change it to RTBC, but do y'all know what's left needing work, or maybe can we move this to RTBC........?

The `snippet` plugin for sure needs test coverage. (Added the tag.) And the IS here describes a different approach. I would be in favor of updating the IS to describe the `snippet` plugin. So I added that tag too.

I am updating the title and issue summary, since I think we have consensus on using the snippet plugin.

To do:

1. Convert the patch to a MR.
2. Add tests.

I am assigning the issue to myself, but if I do not make any progress in a week, then feel free to change that.

I made a MR based on the patch in #19.

I notice that there is a todo comment in the code:

    // @todo Add error checking: the module is enabled, the file exists, the
    // YAML is valid.

We should handle that as well as add a test.

I handled the @todo comment and added a kernel test. This issue is ready for review.

I've been working on the "T" part of RTBC and almost everything is amazing. I'm getting helpful exception messages when the snippet of my yaml is bad or when the "path" I configured was incorrect. Or when I forget to put the "module" or when I use a module that isn't installed (which is even a different message!). Really awesome DX.

I've also used the plugin successfully on multiple projects.

The only problem I've noticed is that the snippet plugin doesn't attempt to handle stopping a pipeline. (#3245997: Allow process plugins to stop further processing on a pipeline)

What should happen if I use this snippet?

- plugin: callback
  callable: is_numeric
- plugin: skip_on_empty
  method: process
- plugin: default_value
  default_value: 'should I ever see this?'

I think I'd want to get null but instead I get that default message when the source is not numeric.

If we don't support stopping in a snippet, we should document that. I think the way to support it would be to go back to the foreach loop and do something like:

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    foreach ($this->pipeline as $step) {
      $value = $step->transform($value, $migrate_executable, $row, $destination_property);
      if ($step->isPipelineStopped()) {
        break;
      }
    }
    return $value;
  }

@danflanagan8:

Thanks for all those good ideas! I think the plugin will be a lot more robust thanks to this input. Back to NR.

Since isPipelineStopped() was introduced in Drupal 10.3.0, and migrate_plus declares compatibility with 9.1, I added method_exists($step, 'isPipelineStopped') to the test. With earlier versions of Drupal core, the code will never reach the break line, but a MigrateSkipProcessException will be handled by executable.

I am disappointed: I do not think I have ever used array_reduce() before, and I thought it was an elegant way to implement transform(). But I do not know how to break out of array_reduce(), except by throwing an exception, so I went back to a foreach loop.

On second thought, I need to do more: the snippet plugin should implement isPipelineStopped(), not just check that method for the process plugins in the pipeline. Maybe this is a hint that I should change the implementation to be more like the sub_process plugin, so that I am not re-implementing the executable. If I do that, then using source: in a snippet should work normally.

I am setting the status to NW and assigning this issue to myself. I intend to update it within the next few days.

On second thought, I need to do more: the snippet plugin should implement isPipelineStopped()

Oh, this is actually a really subtle and interesting point. What IS the snippet? I think I was imagining that stopping within the snippet pipeline would not result in the main pipeline stopping. Does one imagine the snippet as a branch off of the main pipeline (like subprocess)? Or does one imagine the snippet as being inserted into the main pipeline (basically a syntactic trick)? I think there are good uses for both implementations.

Maybe this is a hint that I should change the implementation to be more like the sub_process plugin, so that I am not re-implementing the executable.

It depends on what we're going for. Is the snippet supposed to act more like subprocess? Or are we supposed to imagine the snippet's yaml being inserted into the main pipeline?

If I do that, then using source: in a snippet should work normally.

That occurred to me too! Your beautiful new bit of documentation would be for naught. :(

It depends on what we're going for. Is the snippet supposed to act more like subprocess? Or are we supposed to imagine the snippet's yaml being inserted into the main pipeline?

Personally, I think of it the second way.

A more important consideration is that the plugin should behave the same way in Drupal 10.2 as it does in 10.3 and 11.0. I added a test: it uses a snippet that includes skip_on_empty. In older versions of Drupal (and with the version of this plugin that several people have been using) that throws a MigrateSkipProcessException exception, which is caught by the code calling the snippet plugin. In order to get the same behavior with newer versions, I stop the pipeline and return NULL.

I did not quite get it to work, but I think the constructor could create a stub migration using the null source and destination plugins and

process:
  snippet_result:
    - plugin: get
      source: snippet_value
    - <insert pipeline here>

Then create a MigrateExecutable object with that migration and save that as a class variable instead of the $pipeline array. Then the transform() method would just have to set snippet_value on the Row object, call MigrateExecutable::processRow(), and return the snippet_result destination property.

I think I will leave well enough alone and set the issue status back to NR.

> Personally, I think of it the second way.

Gotcha. Re-reading the IS, I'm convinced that's what's being asked for anyway.

> A more important consideration is that the plugin should behave the same way in Drupal 10.2 as it does in 10.3 and 11.0.

It should be tested manually, but the latest looks good to me. It's right to return $value instead of NULL, which you must have realized after #37

Leaving at NR for the T part of RTBC, as you say.

Finally back to re-test!

FYI, on my current project, I'm only using snippet to do text format mapping on all my rich text fields (so far) -- so, it's only one snippet, and it's only on one type of field/data/whatever -- and only on one use per migration.
Before today, I was using the 2024-08-01 version of the patch/MR.

• Today, I applied the latest version of the patch/MR.
• I ran one of my "upgrade_d7_node_complete_TYPE" migrations that I'd run previously, with the --update flag -- everything worked great.
• I rolled back that "upgrade_d7_node_complete_TYPE" migration, then re-ran it, just, to simulate doing it "fresh" or something -- everything worked great.
• I did the same thing with one other "upgrade_d7_node_complete_TYPE" migration -- success with that one, too.

"Everything worked great" = node data came in, no errors, proper text formats applied to rich text fields 🎉

Please let me know if there's any other info I should share. Thank you!!

I was worried when I saw additional notes in the doc comments about needing to explicitly `get` source values inside the snippet, because I have code like this:

Parent migration:

...
process:
  name: name
  _parent_id: source/parent/id
  description/value: description/value
  description/format:
    plugin: snippet
    module: sw_migrate_content
    path: text_format
    source: description/format
...

Snippet:

-
  plugin: static_map
  bypass: true
  map:
    uikit: legacy
-
  plugin: if_condition
  condition: not:empty
  else_get: constants/full_html_format
-
  plugin: concat
  source:
    - null # null trick current pipeline value
    - '@_parent_id'

But things APPEAR to still have worked fine, both before (using an earlier version of this patch) and now. Is this comment only for the first step in the pipeline? It is confusing to me.

I was afraid the '@_parent_id' and/or the 'constants/xyz' sources would start to fail.

Unless I'm missing something? The comment in question:

* Normally, any process plugin can specify the source key, which implicitly
* adds the get process plugin to the pipeline. This does NOT WORK inside a
* snippet:
*
* @code
* # The source will be ignored. The source of the snippet plugin is always
* # used.
* - plugin: default_value
* default_value: default value
* source: some_field
* @endcode
*
* Instead, explicitly add the get plugin to the pipeline:
*
* @code
* - plugin: get
* source: some_field
* - plugin: default_value
* default_value: default value
* @endcode

@amaisano:

I tested your example with both the current version of MR 101 and the first commit on the feature branch, which is the same code as Comment #19. Both times, your snippet did not work. Do you have any custom code that might be involved?

I created the test module sw_migrate_content using drush gen module, and I copied your snippet into modules/sw_migrate_content/migrations/process/text_format.yml. I installed the Migrate Conditions and Migrate Sandbox! modules. I copied the process section of the migration from your comment and entered the following for the data row:

source:
  parent:
    id: 17
description:
  value: test description from source
  format: uikit

For the constants, I entered

full_html_format: full_html

When I tried to process the row, I got the error

(Migrate Message) migrate_sandbox:description/format:snippet: 'legacy' is not an array

When I modified the snippet as suggested in the doc block (explicitly adding the get plugin), it worked as expected:

_parent_id: 17
description:
  value: 'test description from source'
  format: legacy17

I do not see any changes in the code that would affect this behavior. I added error checking, used constructor promotion, and updated the code to check isPipelineStopped() when it is defined. None of that seems relevant.

Thanks for the candid feedback and collaboration on this ticket.

Can we add a change record with docs how to use this?