Process plugin: build an array from source, destination, pipeline

In #3236774-9: Provide ability to reference current value of process pipeline as a source property, @danflanagan8 pointed out that passing null as part of the source array of a process plugin has the effect of inserting the pipeline value.

At #3436877: [meeting] Migrate Meeting 2024-03-28 2100Z, @mikelutz said that this behavior is a bug, and we should avoid using it.

The process plugin proposed here is more flexible, and it makes the intention clearer.

I think this process plugin is also more flexible than the one proposed in #3314502: Add wrapper process plugin to wrap/unwrap values in arrays.

Some more examples, from the doc block:

Generic example:

process:
  bar:
    plugin: build_array
    source: foo
    template:
      key: literal string
      properties:
        - source:field_body/0/value
        - dest:field_body/0/value
      - pipeline:some/nested/key

Prepare an entity reference revision (ERR) field

process:
  field_paragraph:
    - plugin: migration_lookup
      # ...
    - plugin: build_array
      template:
        target_id: pipeline:0
        target_revision_id: pipeline:1

Here is an example from my current project.

Create a serialized array for the layout_paragraphs module

process:
  behavior_settings:
    # ...
    - plugin: build_array
      template:
        layout_paragraphs:
          parent_uuid: pipeline:0/value
          region: first
    - plugin: callback
      callable: serialize

@danflanagan8:

Thanks for taking a look!

I added some tests, starting with yours from #3314502: Add wrapper process plugin to wrap/unwrap values in arrays. That way,

1. Your excellent test coverage does not go to waste if we decide to close that issue in favor of this one.
2. I confirm that this plugin can do anything the wrap plugin can do (with method: wrap).

When I first wrote this plugin, I called it array_build, but then I realized the problem. I confess I was not very creative when I turned it into build_array, and you are right about the potential for confusion.

I can go with array_template, but other variations are worth considering:

• array_template
• template_array
• template

Or maybe the name should indicate that it can work with source, destination, and pipeline values.

How do you feel about using the short form "dest" instead of spelling out "destination"?

I changed the plugin ID to array_template, and I changed the class names (plugin and test classes) to match.

Here is a more complicated example from my current project. A custom source plugin provides the source field filters that looks something like this:

[
  [
    'vid' => 'some_vocab',
    'tids' => [1, 2, 3, 5],
  ],
  [
    'vid' => 'another_vocab',
    'tids' => [8, 13, 21],
  ],
]

That represents two vocabularies and a few terms from each vocabulary.

Here is the pipeline:

  field_hwp_default_filter_values:
    - plugin: sub_process
      source: filters
      process:
        data:
          - plugin: array_template
            template:
              target_id: 'pipeline:'
            source: tids
        reference_field:
          - plugin: migration_lookup
            migration: hwp_vocabularies
            source: vid
          - plugin: array_template
            template:
              - field
              - 'pipeline:'
          - plugin: concat
            delimiter: _
    - plugin: single_value
    - plugin: array_template
      template:
        - 'pipeline:'
        - data
        - reference_field
    - plugin: callback
      callable: array_column
      unpack_source: true
    - plugin: callback
      callable: serialize

After the first step in the pipeline (sub_process), the example at the top is converted to this:

[
  [
    'data' => [['target_id' => 1], ['target_id' => 2], ['target_id' => 3], ['target_id' => 5]],
    'reference_field' => 'field_some_vocab',
  ],
  [
    'data' => [['target_id' => 8], ['target_id' => 13], ['target_id' => 21]],
    'reference_field' => 'field_another_vocab',
  ],
]

By default, a process plugin (like array_template) is applied to each element of a source array. In this example, migration_lookup is a no-op.

The single_value process plugin overrides that default behavior, so the next array_template prepares the input for the callback plugin:

[
  [['data' => ..., 'reference_field' => ...], ['data' => ..., 'reference_field' => ...]],
  'data',
  'reference_field',
]

and so the callback plugin returns

array_column(..., 'data', 'reference_field')

or

[
  'field_some_vocab' => [['target_id' => 1], ['target_id' => 2], ['target_id' => 3], ['target_id' => 5]],
  'field_another_vocab' => [['target_id' => 8], ['target_id' => 13], ['target_id' => 21]],
]

The last step in the pipeline uses callback with callable: serialize to serialize that array.

@danflanagan8:

I think I fixed all the things you pointed out on the MR.

I agree that 'pipeline:' is awkward. I even considered allowing pipeline as a synonym, but I think the Migrate API already goes too far in making things "convenient". If I did that, then I would have to add test coverage for it, too. In the end, I decided to keep the PHP simple and accept a little ugliness in the YAML for the sake of consistency.