Migrate on D7 has the concept of migration groups. These have two primary uses:

1. Easily manipulating a set of related migrations together - e.g., drush migrate-import --group=commercesite
2. Storage of configuration shared among related migrations, most commonly source information - migrate_d2d stores database connection info in the group, wordpress_migrate stores the location of the source WXR file, etc.

This will be a good thing to build into the core framework in D8, and to have migrate_drupal group the migrations it manages to separate them from any other migrations that might be instantiated in the environment.

In D7, groups were maintained in the migrate_group table, and each individual migration had a group machine_name in the migrate_status table pointing to its group. I'm not sure that's necessarily best in D8 - perhaps it would be better for each group to have a list of member migrations? This would enable a given migration to belong to more than one group, although the concrete use case for doing that escapes me at the moment...

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

moshe weitzman’s picture

Views config entities have a 'tag'. Might be useful to mimic that here.

benjy’s picture

Project: IMP » Drupal core
Version: » 8.x-dev
Component: Code » migration system
Assigned: Unassigned » benjy

I'm going to have a go at this in the next week to assist with D7 migrations.

benjy’s picture

Status: Active » Needs review
FileSize
44.81 KB

First attempt at this attached.

Few things to discuss:

  1. Do we agree that migrations can be in multiple groups? It will allow us to re-use migrations for D7.
  2. Are we happy with migration_groups as the key name?
  3. Do we want a helper method like Migration::loadByGroup($group_name) ? Something like below:
  function loadByGroup($group) {
    $results = \Drupal::entityQuery('migration')
      ->condition('migration_group.*', $group)
      ->execute();
    return entity_load_multiple('migration', $results);
  }
hussainweb’s picture

1. Do we agree that migrations can be in multiple groups? It will allow us to re-use migrations for D7.

Drupal 7's migrate only allows a migration to be in a single group but it would be a cool thing to support multiple groups. I am just wondering how many use cases there would be for that.

2. Are we happy with migration_groups as the key name?

How about just groups?

3. Do we want a helper method like Migration::loadByGroup($group_name) ? Something like below:

It sounds like a good idea.

hussainweb’s picture

To elaborate on the second point: None of the other keys are prefixed. We could keep it consistent and call it just groups without any prefix.

About point 2 in the issue:

2. Storage of configuration shared among related migrations, most commonly source information - migrate_d2d stores database connection info in the group, wordpress_migrate stores the location of the source WXR file, etc.

Should this be implemented as a new config entity, perhaps? If we go this route, then maybe we should follow D7 migrate's approach with using a machine name for group (also would help the drush command line) and more details in the group's yml.

benjy’s picture

1. Many of the D7 migrations are going to be quite similar to the D6 ones.

2. Not exactly true, we have migration_dependencies because "dependencies" conflicted with a parent property.

Yeah, I don't mind the idea of using a machine name instead, that doesn't have to be 1 to 1 with a new config entity. Not sure about the new config entity, I guess we'd have to do that if we want to be able to store configuration in the group as the issue summary states?

chx’s picture

Status: Needs review » Reviewed & tested by the community

This is an awesome first step that can iterate later. If we need to change from Drupal 6 to drupal_6 that's fine because in HEAD grep -- ' - Drupal 6' * in the config directory comes back empty so search/replacing that will target these only.

benjy’s picture

Added a follow-up for the group configuration stuff. #2302307: Support shared configuration between migrate groups

hussainweb’s picture

FileSize
44.01 KB

@benjy: I see your point about migration_dependencies. However, I see to remember an issue that removed prefixes from all (or most of) the properties. Is there a reason we cannot use groups? I am just trying it out to see if there are any obvious failures.

benjy’s picture

The reason I didn't want to use groups is because it's quite generic and if we ever decide to add "groups" in the base config entity like we did with "dependencies" we'd need an API change.

plus, migration_groups is more descriptive, I think I prefer it.

+1 for #3.

hussainweb’s picture

@benjy: Thanks for the explanation. I looked back at the YML files and do see your point. I think groups is descriptive enough, especially when it lives in the migration yml file but the point about base config entity is more important.

alexpott’s picture

Is #3 rtbc or #9? I think #3 - can the real rtbc patch be uploaded last since this is the patch that the rtbc retest get's run against. Thanks.

hussainweb’s picture

FileSize
44.81 KB

Uploading patch from #3 again.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 13: 2202511-3.patch, failed testing.

benjy’s picture

Status: Needs work » Reviewed & tested by the community

Back to RTBC since the last patch isn't the patch that was RTBC

benjy’s picture

FileSize
44.81 KB

O, sorry, cross post. Re-rolled after conflict with #2302261: Typo in migrate.schema.yml

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 16: 2202511-16.patch, failed testing.

hussainweb’s picture

Status: Needs work » Needs review
FileSize
44.86 KB

Fixing conflicts from #2292821: Use widget for comment subject field. It was just a small conflict and I guess it can go back to RTBC as soon as tests pass.

hussainweb’s picture

Status: Needs review » Reviewed & tested by the community

Back to RTBC as per #7.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed 7a42da2 and pushed to 8.x. Thanks!

  • alexpott committed 7a42da2 on 8.x
    Issue #2202511 by hussainweb, benjy | mikeryan: Added Implement...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.