Migrating to the video field with the migrate module

I haven't found anything on this, so here's how I'm thinking of solving this.

I've created a video field - field_video - on my d7 site. I could migrate just one video file to it, but turns out I have more useful files that need to end up as part of field_video.

Here's a crude mapping showing what I'm trying to do:

 
* field_source is my source video, on Amazon S3.
* field_playablefile is the transcoded (via Zencoder) source file, on Amazon S3.
* field_thumbnail is a thumbnail for the video.

I could get Zencoder to generate the latter 2 fields again, but I have hundreds of videos, and the thumbnails have been painstakingly done manually over many years, and look better than any auto-generated thumbnail. So that's not an option.

I wrote a class for handling the d6 S3 textfield url to d7 file entity: MigrateExtrasFileS3. I'll contribute it to the Migrate Extras module, since it already has the MigrateExtrasFileYoutube class.

To handle the 3 migrates files at once, I'd need to extend the file field handler. I don't know how to do that, so I'm thinking of migrating just field_source to field_video, then in complete() handle the other two files, and figure out how to add them to field_video in the entity, and re-save the entity.

I will take a look at writing a Video field handler, too.

Is this the right place to have this discussion? Would it be better in the Migrate issue queue, or the Migrate Extras issue queue?

Thanks Jorrit. I think Migrate Extras have recently changed their policy - they want support for contrib module migrations to be in the contrib module itself. From the project page:

The best place to implement migration support for a contributed module is in that module, not in the Migrate or Migrate Extras modules. That way, the migration support is always self-consistent with the current module implementation - it's not practical for the migrate modules to keep up with changes to all other contrib modules. We are no longer adding support for other modules to Migrate Extras.

The Date Migrate module is a good example. It exists as a module within Date and can then be enabled specifically for a migration, and disabled afterwards. I've started writing VideoMigrateFieldHandler to handle video field migrations - let's see if it works.

@kennywyland, yes I did - I just need to do a bit more work on it to get it a bit more stable, and then I'll contribute a patch. In the meantime you can do some preparation for using it. Basically the VideoMigrateFieldHandler doesn't migrate an individual source video directly to the video field - that must be done first and created as a file entity. Once the file entity exists in d7, VideoMigrateFieldHandler then links that field entity to the video field.

In my case, my source video (on d6) is a youtube url stored in field_embedvideo_embed. I migrate it like so:

<?php
/**
 * Migration class for media_youtube entities.
 */
class MediaVideoYoutubeMigration extends diMigration {
  public function __construct() {
    parent::__construct();
    $this->description = t('Migration of Youtube videos for Video nodes');

    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'uri' => array(
          'type' => 'varchar',
          'length' => 255,
          'not null' => TRUE,
          'description' => 'YouTube URI',
        )
      ),
      MigrateDestinationMedia::getKeySchema()
    );

    // We are getting data from tables in the Drupal default database - first,
    // set up a query for this data.
    $query = Database::getConnection('default', 'd6')
      ->select('content_type_video_embed', 'v');
    $query->addField('v', 'field_embedvideo_embed', 'uri');
    $query->condition('v.field_embedvideo_embed', '%youtube%', 'like');
    $query->orderBy('v.nid', 'ASC');

    $source_fields = array(
      'uri' => t('URI of a YouTube video'),
    );

    $this->source = new MigrateSourceSQL($query, $source_fields, NULL, array('map_joinable' => FALSE));

    // In this case, we need to specify the file_class in the second paramter -
    // this class understands how to translate a http://www.youtube.com/ URI
    // into Drupal's Youtube file scheme (youtube://).
    $this->destination = new MigrateDestinationMedia('video',
      'MigrateExtrasFileYoutube');

    // We just need to map the 'value' in the media destination to the Youtube
    // URI.
    $this->addFieldMapping('value', 'uri');
  }
}
?>

 
A drush mi MediaVideoYoutube migrates all youtube videos.

The magic comes when doing the main content migration. Let's say your content type is called "Video" and your destination video field is called "field_video" - then for the Video migration you'll add the following field mapping:

<?php
    $this->addFieldMapping('field_video', 'video')
         ->sourceMigration('MediaVideoYoutube');
    $this->addFieldMapping('field_video:file_class')
         ->defaultValue('MigrateFileFid');
?>

Then in prepareRow() (or your migration class database query) you make sure that the value of video is one of the YouTube videos already migrated. The Migrate module sees that the source migration is MediaVideoYoutube and looks there to find the fid of the file entity.

Obviously this last part will only work once the VideoMigrateFieldHandler is working.

The Video field uses a number of files, in the following structure:

* field_video['und'][0]['fid'] is the source video file. If no playable files exist, this one is played.
* field_video['und'][0]['thumbnailfile'] is a file object - the thumbnail chosen for the video.
* field_video['und'][0]['playablefiles'] is an array of file objects. Each one is a potentially playable video file, each created by a transcoding process (such as Zencoder or ffmpeg).

I have values for all of the above which I need to migrate to d7 from d6.

So far I've managed to get the Migrate module to link field_video['und'][0]['fid'] to an already migrated file entity (migrated by class MediaVideoYoutube), by using the following field mapping:

<?php
    $this->addFieldMapping('field_video', 'video')
         ->sourceMigration('MediaVideoYoutube');
    $this->addFieldMapping('field_video:file_class')
         ->defaultValue('MigrateFileFid');
?>

But now how can I get Migrate to look in other migrations for the thumbnail file and the playable files? Here's one solution - define some additional arguments to VideoMigrateFieldHandler, like so:

<?php
    $this->addFieldMapping('field_video', 'video')
         ->sourceMigration('MediaVideoYoutube');
    $this->addFieldMapping('field_video:file_class')
         ->defaultValue('MigrateFileFid');
    $this->addFieldMapping('field_video:thumbnail_migration')
         ->defaultValue('MediaImageThumbnail');
    $this->addFieldMapping('field_video:playablefiles_migration')
         ->defaultValue('MediaVideoPlayablefiles');
?>

...and then create migrations for MediaImageThumbnail and MediaVideoPlayablefiles, and run those before the above migration.

That would mean writing some db_selects in the VideoMigrateFieldHandler. I don't know if there's a way of defining additional sourceMigrations so that the db_selects aren't necessary?

Find attached a patch for the VideoMigrateFieldHandler. It still needs work, but I figured if anyone is going to be looking at #6 and giving input, it would be helpful to see how I'm approaching this.

The Video field does not allow for the direct saving of playable files. It links the playablefiles to the video field on a node load by checking if there are any jobs in the transcoding queue which are finished. So the way to migrate playable files is to pretend that we've added a transcoding job and that the job has already run. The playablefiles will be assigned to the field the next time that the node is loaded.

This is how the Video module links a transcoded playable file to the node:

<?php file_usage_add($file, 'file', $video->entity_type, $video->entity_id); ?>

So, we need access to $video->entity_type and $video->entity_id - which we don't have in function prepare() in VideoMigrateFieldHandler. So let's handle this in prepareRow() - which I'm hoping field handlers support...

Great, thanks mikeryan - I was hoping it would be as easy as that! I've reworked VideoMigrateFieldHandler so that the thumbnail migration now works - see attached.

The next thing to solve is linking the playable files to the video field. These aren't directly created by giving some fid's to the Video field handler, unfortunately. The video module links the playablefiles to the video field on a node load by checking if there are any jobs in the transcoding queue which are finished. So the way to migrate playable files is to pretend that we've added a transcoding job and that the job has already run. The playablefiles will be assigned to the field the next time that the node is loaded.

But...where to add a transcoding job? We can only add it once we know the entity id, which we don't have access to in VideoMigrateFieldHandler (and I've tried running a prepareRow() in that handler, but that doesn't run). So I don't think there's any way of getting VideoMigrateFieldHandler to add a (fake) transcoding job.

The only thing I can think of is to add the transcoding job through a database query in prepareRow() of the Video node migration. Is there a better way?

Here is the specific code (from Transcoder.inc) which links playable files to the video file:

<?php
        // add files to file_managed table and add reference to the file_usage table
        foreach ($item['playablefiles'] as $file) {
          file_usage_add($file, 'file', $video->entity_type, $video->entity_id);
          $output_vid = array(
            'vid' => $video->vid,
            'original_fid' => $video->fid,
            'output_fid' => $file->fid,
            'job_id' => !empty($file->jobid) ? $file->jobid : NULL,
          );
          drupal_write_record('video_output', $output_vid);
        }
?>

I've given up trying to migrate the playable files as part of VideoMigrateFieldHandler. I'm rather going to give it a go in complete() of my main migration.

In the meantime, here's the (working) VideoMigrateFieldHandler patch. To use it, call it like so from your migration:

<?php
    $this->addFieldMapping('field_video', 'video')
         ->sourceMigration('MediaVideoYoutube');
    $this->addFieldMapping('field_video:file_class')
         ->defaultValue('MigrateFileFid');
    $this->addFieldMapping('field_video:thumbnail', 'thumbnail')
         ->sourceMigration('MediaVideoThumbnail');
?>

If you're passing multiple values of video and thumbnail then just pass those values as an array (I assign those values in prepareRow).

If you get the following error, you're not using a source migration which contains the already migrated file entity: WD node: MigrateException: array_flip(): Can only flip STRING and INTEGER values!

Thumbnails may be not viewable if storing the images on a CDN and using the Video thumbnail image formatter on drupal 7.20 or later - for more details see #2011674: Video Thumbnail formatter breaks with images stored on S3 due to security fix in Drupal core update 7.20