I've started work on the 2.0 branch that should allow for more flexible building of configurations. Here are the changes that I see that need to take place on the database level in media_mover_api.module

* rename media_mover_configurations to media_mover_steps - This helps clarify that each of the configurations steps is stored as a specific row

* rename media_mover_steps.verb to media_mover_steps.step - This moves away from the concept of "verb" and to steps for each part of the configuration

* add media_mover_steps.name - Gives each step in the configuration a specific name

* rename media_mover_config_list to media_mover_configurations - The current name is not very descriptive of what the table does.

CommentFileSizeAuthor
#10 mm_upgrade.tgz4.11 KBdarrick
#5 428854.patch3.4 KBdarrick
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

arthurf’s picture

* Rename media_mover_files to media_mover_file_data
* Add media_mover_files
* Move each media mover files to separate rows in the media_mover_files table

mfb’s picture

Subscribe

arthurf’s picture

Some additional items
* generate sids for steps
* generate cids for configurations
* build the step maps for each of the existing configurations
* strip out extra file data

Mark Trapp’s picture

Tagging.

darrick’s picture

FileSize
3.4 KB

I didn't get far with trying to update. But I did start writing a patch.

A couple things missing from above:
* ALTER TABLE media_mover_steps ADD `sid` VARCHAR( 255 ) NOT NULL COMMENT 'Step machine name ID' FIRST
* ALTER TABLE media_mover_steps ADD `action_id` VARCHAR( 255 ) NOT NULL AFTER `module`
* ALTER TABLE media_mover_steps CHANGE `configuration` `settings` MEDIUMTEXT CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL
* Add media_mover_step_map

arthurf’s picture

@darrick thanks for starting to work on this. Here is some proto-code for doing this programatically to make sure that the correct data gets moved over. I think this could serve as the basis for getting this done

  $old_configurations = media_mover_api_configurations_get();
  // Get rid of the old schema
  db_query('DROP TABLE {media_mover_configurations}');
  // Create the new version of the configuration
  db_create_table($ret, 'media_mover_configurations', $schema['media_mover_configurations']);
  // Now we start creating the new configurations
  foreach ($old_configurations as $old_configuration) {
    // First create the shell of the configuration
    $configuration = new media_mover_configuration();
    foreach ($old_configuration as $key => $value) {
      $configuration->{$key} = $value;
    }
    // Get a unique cid
    $configuration->cid = media_mover_api_machine_name_create($old_configuration['name']);
    $configuration->name = $old_configuration['name'];
    
    // Now we need to create the steps from this configuration.
    // * Make a steps for each old step:
    // $steps = array('harvest', 'process', 'complete', 'store');
    // foreach ($old_steps as $step_order => $old_step) {
    //   $step = new media_mover_step();
    //   foreach ($old_step as $key => $value) {
    //     $step->{$key} = $value;
    //   }
    // * Get a unique sid for the step
    // * Create the settings (this is going to be rough because we have to map
    //   some of the new step names to old ones
    // * Save the step
    // $step->save();    
    // * Now add each step to the configuration
    // $configuration->steps[$step_order] = $step;
    
             
    // Now we need to create the step map from this configuration
    // Map the sid and cids
    // Insert into db

    // Now we can save the configuration
    $configuration->save();
    
    // Now we can start handling the files
    $old_files = media_mover_api_files_get($old_configuration['cid']);
    // Get rid of the old schema
    db_query('DROP TABLE {media_mover_files}');
    // Create the new version of the configuration
    db_create_table($ret, 'media_mover_files', $schema['media_mover_files']);
    foreach ($old_files as $old_file) {
      $file = new media_mover_file();
      $file->cid = $configuration->cid;
      foreach ($old_file as $key => $value) {
        $file->{$key} = $value;
      }
    // * Figure out if there is any data to save
    // * Now save the file
    // $file->save();
    }
    
  }
darrick’s picture

@arthurf I've updated your code and ended up with this. MM_OMP is my own module.

<?php



define('MM_OMP_HARVEST_DIR', 1);
define('MM_OMP_HARVEST_CCK', 2);
define('MM_OMP_STORE_CCK',3);
define('MM_OMP_COMPLETE_CCK', 4);
define('MM_OMP_PROCESS_THUMBNAIL', 5);
define('MM_OMP_PROCESS_VIDEO', 6);
define('MM_OMP_PROCESS_H264', 7);
define('MM_OMP_STORE_CCK_FLASH', 8);
define('MM_DIR_HARVEST_ACTION', 1 );
define('MM_DIR_STORAGE_ACTION', 2 );
define('MM_DIR_COMPLETE_ACTION', 3 );

  $actions =  media_mover_api_actions_get(MMA_ACTION_TYPE_BOTH);
  $action_map = array(
    'mm_omp' => array(
      MM_OMP_HARVEST_CCK => 'harvest',
      MM_OMP_STORE_CCK => 'store_cck',
      MM_OMP_COMPLETE_CCK => 'store_cck',
      MM_OMP_PROCESS_THUMBNAIL => 'process_video_thumbnail',
      MM_OMP_PROCESS_VIDEO => 'process_video',
      MM_OMP_PROCESS_H264 => 'process_video_h264',
      MM_OMP_STORE_CCK_FLASH => 'store_flash_cck',
    ),
    'mm_dir' => array(
      MM_DIR_HARVEST_ACTION => 'select',
      MM_DIR_STORAGE_ACTION => 'store',
      MM_DIR_COMPLETE_ACTION => 'store',
    ),

  );
  $ret = array();

  // Move all our data to temporary tables the new MM API understands.
  $ret[] = update_sql("RENAME TABLE  {media_mover_configurations} TO  old_media_mover_steps");
  $ret[] = update_sql("ALTER TABLE  old_media_mover_steps CHANGE  `verb`  `step` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL");
  $ret[] = update_sql("ALTER TABLE  old_media_mover_steps ADD  `sid` VARCHAR( 255 ) NOT NULL COMMENT  'Step machine name ID' FIRST");
  $ret[] = update_sql("ALTER TABLE  old_media_mover_steps ADD  `name` VARCHAR( 255 ) NOT NULL COMMENT  'Name of the step' AFTER 'sid'");
  $ret[] = update_sql("ALTER TABLE  old_media_mover_steps ADD  `action_id` VARCHAR( 255 ) NOT NULL AFTER  `module`");
  $ret[] = update_sql("ALTER TABLE  old_media_mover_steps CHANGE  `configuration`  `settings` MEDIUMTEXT CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL");
  $ret[] = update_sql("RENAME TABLE  {media_mover_config_list} TO  old_media_mover_configurations");
  $ret[] = update_sql("RENAME TABLE  {media_mover_files} TO  old_media_mover_files");
  // Load the old configurations with the new api.
  //
  // Create a db prefix so we can use the new api to pull data from the old tables.
  global $db_prefix;
  $current_db_prefix = $db_prefix;

  $temp_db_prefix = array(
    'media_mover_configurations' => 'old_',
    'media_mover_files' => 'old_',
    'media_mover_steps' => 'old_',
  );

  if (is_string($current_db_prefix)) {
    $temp_db_prefix['default'] = $current_db_prefix;
  }

  $db_prefix = $temp_db_prefix;
  $old_configurations = media_mover_api_configurations_load();
  $db_prefix = $current_db_prefix;

  // Create the new tables.
  module_load_install("media_mover_api");
  $schema = media_mover_api_schema();
  db_create_table($ret, 'media_mover_configurations', $schema['media_mover_configurations']);
  db_create_table($ret, 'media_mover_files', $schema['media_mover_files']);
  db_create_table($ret, 'media_mover_step_map', $schema['media_mover_step_map']);
  db_create_table($ret, 'media_mover_steps', $schema['media_mover_steps']);
  db_create_table($ret, 'cache_media_mover', $schema['cache_media_mover']);

  // Create the new version of the configuration
  foreach ($old_configurations as $old_configuration) {

    // First create the shell of the configuration
    $configuration = new media_mover_configuration();
    foreach ($old_configuration as $key => $value) {
      $configuration->{$key} = $value;
    }

    // Get a unique cid
    $configuration->cid = media_mover_api_machine_name_create($old_configuration->name);
    $configuration->name = $old_configuration->name;
    
    $results = db_query("SELECT * FROM old_media_mover_steps WHERE cid=%d",$old_configuration->cid);

    $old_steps = array();
    while ($old_step = db_fetch_array($results)) {
      if ($old_step['module'] == 'media_mover_api') {
        continue;
      }
      $old_step['settings'] = unserialize($old_step['settings']);
      $old_steps[$old_step['step']] = $old_step;
    }

    // Old steps in the step_order we want them converted to.
    $steps = array('harvest', 'process', 'storage', 'complete');
    $step_order = 0;

    unset($last_source);

    foreach ($steps as $action_id => $old_action_id) {
      if (!isset($old_steps[$old_action_id])) {
        continue;
      }
      $old_step = $old_steps[$old_action_id];
       $new_step = new media_mover_step();
       $new_step->settings = $old_step['settings'];
       $action = $action_map[$old_step['module']][$old_step['action']];
       $build = $actions[$old_step['module'] . '--' . $action];
       if (!$build) {
         drupal_set_message(t("Unable to migrate step for %action configuration of module %name.",array('%action' => $old_action_id, '%name' => $old_step['module'])));
         continue;
       }
       $new_step->sid = media_mover_api_machine_name_create($build['description'], 'step');
       $new_step->name = $build['description'];

       $new_step->build = $build;
       $configuration->steps[$step_order] = $new_step;

       // Migrate the files for this step.
       //
       $db_prefix = $temp_db_prefix;
       $old_files = media_mover_api_files_get($old_configuration->cid);
       $db_prefix = $current_db_prefix;
       foreach ($old_files as $old_file) {
         $file = new media_mover_file();
         $file->cid = $configuration->cid;
         $file->source_filepath = $last_source ? $last_source : $old_file->{$old_action_id . "_file"};
         $file->filepath = $old_file->{$old_action_id . "_file"};
         $last_source = $file->filepath;
         $file->step_order = $step_order;
         // * Figure out if there is any data to save
         // * Now save the file
         $file->save();
       }
       $step_order++;
    }
    // Now we can save the configuration
    $configuration->new = TRUE;
    $configuration->status = 'enabled';
    $configuration->save();
    
  }
?>
arthurf’s picture

This looks good to me- do you want to patch media_mover_api.install so we can have other people start testing this?

And, thanks again for all the patches and the effort- it's *great* to get some support

darrick’s picture

I can start working on that. One option I was thinking about other then patching the install was to create a seperate mm_migration module. As it seems like a awful lot of work for the update to be doing.

Updating the db and backing up the tables could be done in install but the migration of configurations might be better in the mm_migration. I would imagine the migration would take a very long time and a separate module would allow it to be done in batches of 100 or so. I could also see issues crop up where the user would need to be asked how best to migrate the configurations.

Either way with all the patches I've done the past few days I do have a better understanding or the new API. One thing I'm unsure about is how to fill the data array for the new files. My code above was just splitting the old mm_file into 4 mm_files. One for each step. I don't think this is correct. I should be taking the one file and then creating a data array for it for the different steps it has gone through. I have no clue what to do if the filename has changed. Do I just create a new file entry with the old_name as source_filepath and the new name as filepath and then continue the steps in this new entries data field?

BTW: I'm happy to help. My goal is see v2 get done because I would like to see a version for D7.

darrick’s picture

Version: 6.x-1.x-dev » 6.x-2.x-dev
FileSize
4.11 KB

I've ended up with a separate module for migration. My reasoning being:

- Easier to debug the migration. You don't have to keep resetting mm db_version or manually remove broken configurations previously migrated.
- I put in two drupal_alter functions:
-- One to alter the data of the old_configurations. i.e. I'm splitting up older storage step into a new storage and complete steps.
-- The other is to alter the build array when mapping old steps to new steps. i.e. I'm moving some of my custom steps to the newer mm_node steps.

Here's some examples of how I'm using the hooks:

function mm_omp_media_mover_old_configurations_alter(&$old_configurations) {
  // Implementation specific to DMA
  // Map the old action to the new action.
  foreach ($old_configurations as $key => $old_configuration) {
    $module = $old_configuration->storage->configuration['module'];
    $action = $old_configuration->storage->configuration['action'];

    if ($module == 'mm_omp' && $action == '3') {
      $old_configurations[$key]->complete = $old_configuration->storage;
    }
  }
  // \DMA
}

define('MM_OMP_HARVEST_DIR', 1);
define('MM_OMP_HARVEST_CCK', 2);
define('MM_OMP_STORE_CCK',3);
define('MM_OMP_COMPLETE_CCK', 4);
define('MM_OMP_PROCESS_THUMBNAIL', 5);
define('MM_OMP_PROCESS_VIDEO', 6);
define('MM_OMP_PROCESS_H264', 7);
define('MM_OMP_STORE_CCK_FLASH', 8);

function mm_omp_media_mover_map_actions_alter(&$build, $old_configuration, $verb) {
  if (empty($build)) {
    $actions =  media_mover_api_actions_get(MMA_ACTION_TYPE_BOTH);

    // TODO: Add the switches for the rest of the submodules.
    $action_map = array(
      'mm_omp' => array(
        MM_OMP_HARVEST_CCK => 'harvest',
        MM_OMP_STORE_CCK => 'store_cck',
        MM_OMP_COMPLETE_CCK => 'store_cck',
        MM_OMP_PROCESS_THUMBNAIL => 'process_video_thumbnail',
        MM_OMP_PROCESS_VIDEO => 'process_video',
        MM_OMP_PROCESS_H264 => 'process_video_h264',
        MM_OMP_STORE_CCK_FLASH => 'store_flash_cck',
      ),
    );

    // Map the old action to the new action.
    $module = $old_configuration->{$verb}->configuration['module'];
    $action = $old_configuration->{$verb}->configuration['action'];

    $new_action = $action_map[$module][$action];


    $build = $actions[$module . '--' . $new_action] ? $actions[$module . '--' . $new_action] : array();
  }
  // Implmentation specific to DMA
  if ($build['action_id'] == 'store_cck' && $build['module'] == 'mm_omp') {
    if ($verb == 'complete') {
      $build['action_id'] = 'save_as_node';
      $build['module'] = 'mm_node';
    }
    else if($verb == 'storage') {
      $build['action_id'] = 'extract_showid';
      $build['module'] = 'mm_omp';
    }
  }
  // \DMA
}

- Currently there is a break in the recursion for migrating files. So only one file is migrated per conversion.
- The module only migrates files which are in the complete state.
- I found hook_media_mover_file_save_alter useful for cleaning up my filepaths during migration.
- I found hook_media_mover_step_save_alter useful for mapping some of the older step settings to the new step settings.