Hello, folks.

I'm doing an interesting migration from a MS SQL database to a Drupal 6 database using the migrate module, and I want to make sure that I'm doing things in the general way that you folks intend the module to be used. I'd hate to find out that I've been doing things the hard way, after all.

My major problem is that I'm going to be pulling source data from several similar MS SQL databases to the same Drupal site, and so when I saw beta3 come out with arguments and dynamic migration classes, it seemed to be the solution I needed - copy the migration classes (4 so far) to a second set that included arguments that indicated a second database to read from. I also overrode the isDynamic() function to return TRUE.

My first trouble is that it seems that there's a break in how migration names are handled - although you specify the migration name in the machine_name of the migrate_status table, this doesn't get passed to the class when it's created, so when parts of the system ask the class what it's migration name is, it gets back the BarFoo for class BarFooMigration, even if I've put a migration name of BarFooBig in migrate_status. Should I pass in the migration name as one of the arguments?

My second question/trouble is for dependencies. Naturally I've got BarFoo2 dependent on BarFoo1 being run, but what about BarFooBig2 and BarFooBig1 that I've declared in migrate_status? It seems a little messy to have to write code inside of the migration classes to compare $this->machineName and figure out what the dependencies are supposed to be, or is this another thing I should pass in as an argument?

Thanks.

Comments

spinoza_gl’s picture

spinoza_gl commented

For what its worth, and this from currently trying to migrate data from a heavily normalized MS SQL database,
flatten your data/denormalize as much as you can. What you can't denormalize, read it in in the prepareRow function.

In retrospect, ETL from MS SQL to MySQL seems to have been the comparatively easier part. It is stuffing data into drupal's schema that is challenging. Not sure if this helps and FYI, I am doing this with drupal 7 rc 4.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Title: Dynamic Migration Classes - Am I doing this right? » Document how to use dynamic migrations
Component: Miscellaneous » Documentation
Category: support » task

Let's make this a documentation issue, I need to fill in http://drupal.org/node/1006998. I don't have time to fully explain it right now, but the essence is as each distinct source to be passed through a common set of Migration classes is identified, generate unique machine names and call MigrationBase::registerMigration($class_name, $machine_name) (one-time process). You need to define generateMachineName($class_name) to construct machine names from the class name and whatever unique data you have for the source. As for specifying dependencies (and sourceMigrations), do something like

    $this->dependencies = array($this->generateMachineName('MyBlogEntry'),
      $this->generateMachineName('MyPage'));

For a concrete example (the one for which this capability was developed), see this module (not yet ready for prime-time, but should serve well as an example to follow).

jcfiala’s picture

jcfiala commented

Ah, so I am doing things correctly, on the whole. Excellent.

Actually, I did up this in my base migration class:

protected function getProperMachineName($names) {
    // If the base Machine name is the same as the machine name, then just return the input.
    if ($this->baseMachineName == $this->machineName) {
      return $names;
    }
    
    // if we're given a value other than an array, stick it in an array.
    $is_array = TRUE;
    if (!is_array($names)) {
      $is_array = FALSE;
      $names = array($names);
    }
    
    // Get suffix for names:
    $suffix = str_replace($this->baseMachineName, '', $this->machineName);
    if ($suffix) {
      $result_array = array();
      foreach ($names as $name) {
        $result_array[] = $name . $suffix;
      }
    }
    
    if ($is_array) {
      return $result_array;
    }
    else {
      return $result_array[0];
    }
  }

Where baseMachineName is a protected variable set to the machineName before I change it to the proper machine name.

jcfiala’s picture

jcfiala commented

Having had a look at it, Mike, I suggest putting a link to the wordpress migration module as an additional reference on the migration project page after the link that goes to the Economist page, with a note that it shows doing a dynamic migration and is incomplete... although I suspect anyone going to the project page will see that it's incomplete.

That's a good basis for reference for my code, and I'm currently modifying my migration code to be in line with what you were doing... although happily I was guessing well enough that it's not a huge change.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

I will link that module prominently from the Migrate project page once it's a bit more mature - for the moment, in its pre-alpha state, I prefer to keep it on the DL. But I'm glad you're finding it helpful!

moshe weitzman’s picture

moshe weitzman commented
Status: Active » Fixed

I committed a migrate_example_baseball module (7.x) which uses dynamic migration. I think it uses the right approach. Feedback welcome.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.