[META] farmOS 2.x Contrib Migrations

For lack of a better place (and lack of time), pasting some notes I wrote directly in the farm_migrate_asset migration - relevant to the contrib migration considerations...

Context: while testing asset migrations, and specifically the parent asset reference field, some assets were failing because the stub that was created for them didn't have a bundle set. Setting default_value: stub allowed the migration to work, but made me
realize flaws in the approach/goals:

process:
  # Static map old asset types to new.
  # This only handles known asset types provided by farmOS core.
  # If the asset type is not found in this map, then the asset will be skipped.
  # Contrib farmOS modules are responsible for migrating their own asset types.
  type:
    plugin: static_map
    source: type
    map:
      animal: animal
      compost: compost
      equipment: equipment
      group: group
      planting: plant
      sensor: sensor
    bypass: FALSE
    # Hmm. This default_value is at odds with our overlapping needs.
    # - Without this default_value, the parent field migration tries to create
    #   a stub but fails at this process because the type source is not set
    #   during stub creation.
    # - With this default_value, `bypass: FALSE` has no effect. Which means
    #   all assets will be imported, and wl have a type of `stub` if their
    #   source type is not in this map.
    # - Alternatively, we could set `bypass: TRUE` and omit `default_value`,
    #   which would result in the creation of assets even if their asset type
    #   does not exist.
    #     - On one hand, this would make contrib asset type migrations easier.
    #     - On the other hand, it means our current approach of "only enable
    #       modules for the data you want to migrate" is no longer true. All
    #       data will be migrated.
    #         - Come to think of it, we're already breaking this promise by
    #           not having separate migrations for each asset type. There may
    #           be asset types in this map that were not enabled. They would
    #           still be migrated, simply because they are in this map.
    #
    # With all the above in mind, we have two choices:
    # - Accept that we can't promise "only the modules you enable will be
    #   migrated". Migrate all data regardless.
    #     - This may be OK for core asset/log types, but what about contrib? If
    #       a contrib module is not ready for 2.x, then data will be migrated
    #       for it, but it will be orphaned in the db as an unknown bundle.
    # - Take a different approach. Flip this on its head: create separate
    #   migrations for each bundle, and a general migration for common fields,
    #   but run the bundle migrations first, and fill in the common fields
    #   second, only on the entities that were imported by the bundle
    #   migrations.
    #     - How do we set the dependencies on the migrations to accomplish
    #       this?
    #     - The common fields migration can't explicitly list the bundle
    #       migrations as required dependencies, because then all the bundles
    #       would be required.
    #     - Optional dependencies could work, but only for the bundle
    #       migrations we know about in core. Contrib bundles would also need
    #       some way of ensuring that they run first.
    default_value: stub

I have refactored the core farmOS 1.x migrations based on the thoughts in my previous comment:

Take a different approach. Flip this on its head: create separate
migrations for each bundle, and a general migration for common fields,
but run the bundle migrations first, and fill in the common fields
second, only on the entities that were imported by the bundle
migrations.

This approach makes it relatively easy for contrib modules to add their own bundle-specific migrations. They just need to make sure to put them in the appropriate migration_group:

• farm_migrate_asset
• farm_migrate_log
• farm_migrate_taxonomy

If the bundles define any additional reference fields, these should be in separate migrations that are part of the farm_migrate_reference group, to ensure that they run AFTER the bundle migrations.

The last thing to figure out is: right now the reference migrations (eg: asset parent field, and asset and equipment reference fields on logs) have hard-coded lists of bundle specific migrations that they will look for IDs in. This means that if a contrib module adds their own bundle migrations, these also need to be added to the lists in the reference migrations configs.

Not sure what the best way to do that is... setting this issue to "Needs work".

Not sure what the best way to do that is...

One idea might be to provide our own process plugin that extends the migration_lookup class, and makes it possible to search all migrations in a particular group.

So instead of:

process:
  # Look up logs created by type-specific migrations.
  id:
    -
      plugin: migration_lookup
      source: id
      migration:
        - farm_migrate_log_activity
        - farm_migrate_log_harvest
        - farm_migrate_log_input
        - farm_migrate_log_lab_test
        - farm_migrate_log_maintenance
        - farm_migrate_log_medical
        - farm_migrate_log_observation
        - farm_migrate_log_purchase
        - farm_migrate_log_sale
        - farm_migrate_log_seeding
        - farm_migrate_log_transplanting

It might be:

process:
  # Look up logs created by farmOS log migrations.
  id:
    plugin: migration_group_lookup
    migration_group: farm_migrate_log
    source: id

That could be a nice general-purpose process plugin. Maybe worth sketching it up and contributing as a patch to the Migrate Plus module.

OK! I was able to get that working. I created the new process plugin in the farm_migrate module (as farm_migration_group_lookup), but I also opened this issue to explore adding it to migrate_plus upstream: https://gitlab.com/drupalspoons/migrate_plus/-/issues/240

I think we can close this. We're developing migrations in the contrib modules that we maintain, and these serve as examples for other contrib modules. I don't think we need dedicated documentation for this on docs.farmos.org.