Dynamic migrations

In progress... I've got the basic support in Migrate, haven't tried implementing dynamic classes on top of it yet though. But fair warning, once this change goes in existing migration code will need to be changed.

• MigrationBase::getInstance() will take a machine name rather than a class name.
• dependencies and softDependencies will be machine names rather than class names.
• sourceMigration() arguments will be machine names rather than class names.

Basically, in any of these cases, to fix existing code remove Migration from the end of argument you're currently passing.

Committed for D7, to be backported to D6.

OK, that commit is a bit messed up, please don't update to the latest dev release on D7 until futher notice...

OK, the D7 dev branch should be in better shape. Having some trouble with the D6 backport, though...

OK, Moshe points out this still isn't quite working right. Let me explain what the issues are...

Originally, we had a one-to-one correspondence between migrations (as listed on the dashboard or drush ms) and migration classes. We required migration class names to end with Migration, and set the machine name to the class name without the Migration. Hence, the machine name for BeerUserMigration is BeerUser. This issue was about permitting parameterized migration classes, where one class might have multiple instances (listed and run separately). What I tried to do is maintain the existing behavior, while adding the ability to define migrations where the machine name and class name aren't necessarily related (more specifically, any relation is specific to the migration implementation, the migrate module itself doesn't know it). This was a bit of a struggle, because too many areas of the code assumed the class_name=machine_name.'Migration' pattern, but I seemed to be close. The problem is, though, that essentially we're supporting two kinds of migrations - those that follow the pattern, and those that don't. And, given just a machine name, we can't easily tell which kind it is.

So, I'm thinking it's time to ditch the Migration suffix - no longer make it special. The default machine name will be the class name - if you need to do "dynamic" migrations, or want a different machine name, you can implement hook_migrations_alter() (implemented as part of this issue). I think in the long run that will be cleaner, but in the short run the transition may be a bit painful for those of us already using Migrate 2. I expect I can do it without break existing migration code, but you'll find that the Migration suffix now appears where it previously didn't, and you will have to include it in drush commands (i.e., drush mi BeerUserMigration instead of drush mi BeerUser).

Thoughts?

Oh, and again - the current dev D7 branch is broken, using specific migration names in drush commands doesn't work (i.e., you can do drush mi --all but not drush mi BeerUser). If you're on beta2, or pulled from CVS before my 12/10 commit, stick with what you've got until I've got this all worked out.

I've committed changes per comment #7 - Migration is no longer a special suffix, machine names match class names by default, etc. This seems to work fine at the command line (at least better than it did yesterday), but the tests are hosed in a very strange way (running the Migrate tests I get the message "Class 'ImageToolkitTestCase' not found" - I'm not running image tests!) - I will investigate tomorrow.

A reminder that I haven't touched the D6 branch yet - I will backport once D7 is all straightened out.

Grrrr... More problems, I'm going to think out loud here...

So, the original (pre-dynamic) mechanism for finding out what migrations were available was to browse class names from Drupal's code registry for names ending in Migration, create a ReflectionClass for the name, and identify whether it's a non-abstract subclass of MigrationBase. By removing the Migration suffix restriction, we instantiate a ReflectionClass for every class in the registry. Problem: broken classes can appear in the registry (the registry just parses class names from declarations, it doesn't pick up any number of potential things PHP won't like), and even instantiating the reflection object for the class (not an actual instance of the class) can generate errors. When I first ran into this I setup an error handler to catch these, but the error handler doesn't catch fatal errors.

Now, these errors do reflect actual problems in classes elsewhere in the Drupal install. One approach may be to let it go - fix the broken modules. But that leaves the migration code held hostage to other modules - if this were rare, it might be practical, but just in this one environment there are two modules affected; who knows how many others might be out there? And, the approach is inherently a performance concern anyway - it was one thing to narrow down the field of search by naming conventions, but now it's scanning several hundred classes in the registry; not just for Migration classes, but in separate calls for handlers.

So, I'm now thinking we really need to replace auto-discovery of classes with an explicit registration process. Already dynamic migrations need to explicitly add themselves to the migrate_status table - we might as well extend that process for all migrations. So, when you implement a new migration, rather than simply declare the class and clear the cache, you would write an update function to register the migration (and de-register it in hook_uninstall). It is more work, but I think it is a cleaner way to manage the classes. It also will address an issue that's looming for me - being able to override a non-abstract migration class and control whether the overridden class can be used on its own, or be completely effaced by the override.

Or, rather than having a push model (register/de-register in update hooks), we could have a pull model - implement hook_migration and add your valid migrations there, plus hook_migration_alter for fancy stuff.

And keep in mind, we need to deal with this not just for the Migration classes, but the handler classes as well.

Thoughts? Speak now, because I need to deal with this today...

Thanks.

Or, support both push and pull - pushing adds migrations to migrate_status, and migrate_migrations() returns everything in that table plus anything added via a hook...

OK, I've discussed this with Moshe, who was concerned with the additional housekeeping burden on migration implementors. What we've come up with is per-module registration. The fundamental problem with discovering relevant classes is the limited information available in the Drupal code registry - basically, the name and the defining module. We want to narrow the number of classes reviewed for matching for two reasons - performance, and the fact that sometimes registered classes blow up the ReflectionClass constructor. The original means of doing this was by requiring suffixes - Migration for migration classes, Handler for general handlers, FieldHandler for field handlers. That's limiting for the migration implementor, and just plain ugly.

So, let's look at that module column in the registry. If we knew which modules implement migration classes, we could scan only registry entries for those modules. How do we know which modules those are? Well, the simplest way is a hook whereby a module reports, say, what API version of Migrate it's built against. And, it happens that we had such a thing in Migrate 1, which we haven't addressed yet for Migrate 2.

So, that's the current plan - modules implementing migrations will implement

function example_migrate_api() {
  $api = array(
    'api' => 2,
  );
  return $api;
}

Migrate will pick up the names of the modules implementing this hook (and reporting an API version of 2), and filter the registry query on that list.

This is close now - it appears to be working fine in my development environment, but I'd like to do some more testing before committing (probably tomorrow).

Key changes that will be seen in beta3 from this work:

1. All modules implementing migration classes must implement hook_migrate_api() (see migrate_migrate_api() for an example).
2. Migration class names are no longer required to end in 'Migration'.
3. Dependencies and sourceMigrations must now be expressed in terms of machine name rather than class name.
4. MigrationBase::getInstance() now takes a machine name rather than a class name.

The code is committed to CVS, for both D6 and D7, thanks.

Committed this past weekend... No one's screamed yet.

Yes, the implementing class needs to add to the migrate_status table using MigrationBase::registerMigration(). For example, in the blog migration module I'm working on now, when an XML file is uploaded a migration machine name is generated from the blog title and used to register (and subsequently run) the migration.

On the table name, I've had enough refactoring for now:-). The table does track the status of migrations, so the name still does fit, but you're right that optimally migrate_migrations or migrate_registry might be best.

Map tables are created in the destination database by default, but if you pass the connection key (e.g., 'legacy') of your source database as the fourth argument to the MigrateSQLMap constructor you can have them created there. Also be sure that if the map tables in your destination table cannot automatically join to the source query, you set the 'map_joinable' => FALSE option in your source constructor.