Problem/Motivation

The new include migration plugins are a great feature. However, in practice, the single MYMODULE.migrate_shared_configuration.yml gets big and unwieldy really quickly. It's hard to get an overview of what's there, it's hard to see where one include section ends and another begins, and with multiple developers working on migrations, everyone is working on the same file with resulting git conflicts.

Steps to reproduce

Proposed resolution

Change the include plugins to use one YAML file per include plugin, in a specific subfolder, the same way migration plugins work.

This can be done by changing the plugin manager to use YamlDirectoryDiscovery.

For BC, create a class which decorates the old discovery.

Remaining tasks

User interface changes

API changes

Data model changes

Issue fork migrate_tools-3397621

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

joachim created an issue. See original summary.

joachim opened merge request !49

joachim’s picture

joachim commented
Status: Active » Needs review
amaisano’s picture

amaisano commented

Does this only break it up within the current module, or can it be made so that we can reference different module's shared migration file(s)? Would be nice to have cross-module migration shared yml files (like a family of sub modules where the parent module has a common set of shared migration data, and the sub modules that each deal with different parts of the migration have more specific shared migration data).

joachim’s picture

joachim commented

> Does this only break it up within the current module, or can it be made so that we can reference different module's shared migration file

The migration includes are merely plugins -- you can reference any plugin of that type in any migration. That's already the case with the current code.

amaisano’s picture

amaisano commented

So cross-module use of the shared_configuration YAML data is possible w/o this patch? Like so:

/my-migrations
--/modules
--/--/node_migration
--/--/--/node_migration.migrate_shared_configuration.yml (with a "defaults" include definition)
--/--/term_migration
--/--/--/term_migration.migrate_shared_configuration.yml (with a "taxonomy" include definition)
--/--/misc_migration
--/--/--/migrations
--/--/--/--/misc_migration.yaml (with contents below)

# Misc Migration
id: misc_migration
includes:
  - defaults
  - taxonomy
source: ...

?

joachim’s picture

joachim commented

AFAICT, yes. I had a quick look in the code of MigrateIncludeHandler and it's not doing any doctoring to the 'include' value.

amaisano’s picture

amaisano commented

It would be nice if you could still group multiple inside a single YML file, especially when you have multiple override "snippets" that are only a few lines -- creating a separate file for all of them is a bit overwhelming.

migrate_shared_configuration/defaults.yml
- include1:
-   source:
-     constants:
-       a: 1
- include2:
-   source:
-     constants:
-       b: 2
- include3:
-   source:
-     constants:
-       c: 3

migrate_shared_configuration/big_complex_field.yml
- process:
-   field_paragraphs:
-      plugin: ...   
real_migration.yml
includes:
  - include2
  - big_complex_field
amaisano’s picture

amaisano commented

Also, sub-folders under migrate_shared_configuration/ would be nice:

- migrate_shared_configuration/node/bundle_1.yml
- migrate_shared_configuration/node/bundle_2.yml
- migrate_shared_configuration/term/vocab_1.yml
heddn’s picture

heddn
Primary language English
Location Illinois
commented
Status: Needs review » Reviewed & tested by the community

Nice addition.

  • heddn committed 58e60e82 on 6.0.x authored by joachim 
    Issue #3397621 by joachim, heddn: Change migration include plugins to...
heddn’s picture

heddn
Primary language English
Location Illinois
commented
Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

heddn closed merge request !49