Add d8 media migration

I think this will live with the media system right as with the rest of the integrations. We should define what if any is required for 8.3.0 shipping with media. Need input on that.

I think the release manager review tag is warranted for that.

If Media does not have a migration destination and becomes stable, then that moves Migrate Drupal further from being stable. We should be requiring migration destinations for all added storages at this point. They will also help for migrating core image and file data into media entities once they are stable. So, I think we need to have the migration destinations available for Media to be stable (a "must" have on the initiative roadmap).

However, migrations are not a blocker for adding the API to core so long as the module is still in beta or earlier. For media, we agreed that the module itself could considered a stable API addition but experimental otherwise, so long as the module remains marked hidden. The same should apply here. So we can postpone this issue on finalizing the initial API and on finalizing the file and image types (although we may consider incling the migation destinations for files and images in those patches). Marking postponed on the initial entity issue, at least.

I updated the summary with this information.

Thanks @Gábor Hojtsy!

@xjm: thanks! Since this has some farther reaching implications for release management process, continuing this on https://www.drupal.org/node/2825215#comment-11889256

Just a note - a destination plugin does not appear to be necessary for the media entity, I'm doing a project right now with contrib media_entity and the following works just fine on the destination end:

destination:
  plugin: entity:media
  default_bundle: image

I don't think any process plugins would be necessary either - just source and migration plugins for the upgrade path.

What does the Migrate Drupal UI do for example if there are both migrations for image fields (D6/D7) to image fields (D8) and image fields (D6/D7) to media entities (D8??). I definitely agree update and/or migration paths between D8-D8 would be needed if we abandon (AKA deprecate) image/file fields on entities. That is a big decision to make in itself and needs careful planning as to how it is done.

What does the Migrate Drupal UI do for example if there are both migrations for image fields (D6/D7) to image fields (D8) and image fields (D6/D7) to media entities (D8??).

It executes all migration plugins in enabled D8 modules with the relevant migration_tag (Drupal 6 and Drupal 7). So, if media were enabled on D8, as it is now it would execute both - we'd want to have something in the media module that would disable the unwanted migrations (hook_migration_plugins_alter()?).

Actually, I misspoke earlier - there is one desirable process plugin, one to rewrite embedded images (either <img> tags or D7 media JSON data) to <media-entity> tags. I have such a plugin on my current project for the JSON case (although it obviously would need generalization), but the key challenge there for the upgrade path is that this plugin would need to be inserted into the process pipeline for every text field which may contain embedded images. That'll be fun!

On a moment's further thought... Maybe that processing could be hooked into the text field handling.

Oh #4 is potentially good news! Thanks @mikeryan. So I guess part of the scope here is figuring out exactly what we need in core for stable Media, vs. optional/followup.

#9 sounds scary. Interesting problem for sure.

Is this correct tag to add for Migrate?

@mikeryan well, media module does not deal with embedding, does it? :) That would be entity_embed or some other module you are using there. Similar functionality is desired for core but definitely not as part of the media entity module itself at first.

I wrote a blog post about D7 file entities migration to D8 media entities.

I didn't intend to revert the issue summary. Can someone revert the issue summary to https://www.drupal.org/node/2835825/revisions/view/10404661/10405405, please?

I couldn't quite understand from the diff which was the intended revision, but hopefully this is the right one. Thanks @jibran.

Hi,

I have done the migration for files as media entity in drupal 8 with the external json resource..

I think this does not need to be postponed right now, correct?

#2885002: Provide an optional migration path for sites using files/images to media entities covers part of what's listed in the summary here, so I postponed it on this issue, which can add the migration integration itself.

Here is an initial patch to generate media items from existing permanent files. There are dedicated tests for the source and destination plugins, plus a single end-to-end test of the migration.

There's a lot more than needs doing here (as per the IS), but setting to NR to see if Drupal CI approves. IMHO we should consider making this issue a meta and opening child issues for all the sub-tasks.

Hi @phenaproxima,

I've tested your patch or migration and it works like a charm. All of my files are available as media entities now. Thank you. But how can I migrate existing file fields to media fields?

Ciao
Ralf

Hi @rwam -- migrating the fields is going to require a battery of separate migrations. Personally, I'd like to attack those in follow-up issues. I'll see what I can do about opening those issues, coordinating with the rest of the initiative.

Hi @phenaproxima,

based on your patch I've created my own migration path, but I think it could be interesting for someone else. So I create a short presentation for the Drupal User Group Munich and the slides and also my migration-script are online now. The slides are available at http://dev.rwam.de/d8-migrate-file-to-media/ (in german).

I'm with you, an automatic migration is quiet difficult. My approach is to create an new media field for each field which should to migrate and migrate the content from the file field into the new one by using of a drush script. It works for node and paragraph bundles too and you can migrate only those fields which are needed.

The drush script is available at http://dev.rwam.de/d8-migrate-file-to-media/resources/migrate-file-field... I could migrate successfully about 1100 files and field contents on a multisite environment with this script.

Ciao
Ralf

Bumping this to major, I think this blocks deprecation of configurable file/image fields (recognising they'll still need to be on media entities one way or another).

Created a meta issue for this, and opened #2930513: File/image field to media field migration for one follow-up. I agree with getting the file_managed -> media_entity migration done first then doing the rest in follow-ups.

+++ b/core/modules/media/migration_templates/file_media.yml
@@ -0,0 +1,39 @@
+    map:
+      gif: image
+      jpg: image

Does this need a default: file? We could them maybe just specify the images and let everything else fall back.

Hi,

Just tried an images to media migration with the patch #21 and the migration stops after the first element with this message:

$ drush mim file_media
 [warning] call_user_func() expects parameter 1 to be a valid callback, class 'Drupal\media_entity\Entity\Media' not found FieldConfigBase.php:396
 [error]  Error: Call to a member function getSource() on null in Drupal\media\Entity\Media->getSource() (line 162 of /Users/sahaj/Sites/rw84test/web/core/modules/media/src/Entity/Media.php) #0 /Users/sahaj/Sites/rw84test/web/core/modules/media/src/Plugin/migrate/destination/Media.php(27): Drupal\media\Entity\Media->getSource()
#1 /Users/sahaj/Sites/rw84test/web/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php(85): Drupal\media\Plugin\migrate\destination\Media->getEntity(Object(Drupal\migrate\Row), Array)
#2 /Users/sahaj/Sites/rw84test/web/core/modules/migrate/src/MigrateExecutable.php(224): Drupal\migrate\Plugin\migrate\destination\EntityContentBase->import(Object(Drupal\migrate\Row), Array)
#3 /Users/sahaj/Sites/rw84test/vendor/drush/drush/includes/drush.inc(232): Drupal\migrate\MigrateExecutable->import()
#4 /Users/sahaj/Sites/rw84test/vendor/drush/drush/includes/drush.inc(223): drush_call_user_func_array(Array, Array)
#5 /Users/sahaj/Sites/rw84test/web/modules/contrib/migrate_tools/src/Commands/MigrateToolsCommands.php(716): drush_op(Array)
#6 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), 'file_media', Array)
#7 /Users/sahaj/Sites/rw84test/web/modules/contrib/migrate_tools/src/Commands/MigrateToolsCommands.php(313): array_walk(Array, Array, Array)
#8 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->import('file_media', Array)
#9 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/CommandProcessor.php(235): call_user_func_array(Array, Array)
#10 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/CommandProcessor.php(181): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#11 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/CommandProcessor.php(147): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#12 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(404): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#13 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Command/Command.php(264): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Application.php(859): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Application.php(206): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Application.php(125): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /Users/sahaj/Sites/rw84test/vendor/drush/drush/src/Runtime/Runtime.php(112): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /Users/sahaj/Sites/rw84test/vendor/drush/drush/src/Runtime/Runtime.php(41): Drush\Runtime\Runtime->doRun(Array)
#19 /Users/sahaj/Sites/rw84test/vendor/drush/drush/drush.php(64): Drush\Runtime\Runtime->run(Array)
#20 /Users/sahaj/Sites/rw84test/vendor/drush/drush/drush(4): require('/Users/sahaj/Si...')
#21 {main}.
Error: Call to a member function getSource() on null in /Users/sahaj/Sites/rw84test/web/core/modules/media/src/Entity/Media.php on line 162 #0 /Users/sahaj/Sites/rw84test/web/core/modules/media/src/Plugin/migrate/destination/Media.php(27): Drupal\media\Entity\Media->getSource()
#1 /Users/sahaj/Sites/rw84test/web/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php(85): Drupal\media\Plugin\migrate\destination\Media->getEntity(Object(Drupal\migrate\Row), Array)
#2 /Users/sahaj/Sites/rw84test/web/core/modules/migrate/src/MigrateExecutable.php(224): Drupal\migrate\Plugin\migrate\destination\EntityContentBase->import(Object(Drupal\migrate\Row), Array)
#3 /Users/sahaj/Sites/rw84test/vendor/drush/drush/includes/drush.inc(232): Drupal\migrate\MigrateExecutable->import()
#4 /Users/sahaj/Sites/rw84test/vendor/drush/drush/includes/drush.inc(223): drush_call_user_func_array(Array, Array)
#5 /Users/sahaj/Sites/rw84test/web/modules/contrib/migrate_tools/src/Commands/MigrateToolsCommands.php(716): drush_op(Array)
#6 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), 'file_media', Array)
#7 /Users/sahaj/Sites/rw84test/web/modules/contrib/migrate_tools/src/Commands/MigrateToolsCommands.php(313): array_walk(Array, Array, Array)
#8 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->import('file_media', Array)
#9 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/CommandProcessor.php(235): call_user_func_array(Array, Array)
#10 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/CommandProcessor.php(181): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#11 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/CommandProcessor.php(147): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#12 /Users/sahaj/Sites/rw84test/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(404): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#13 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Command/Command.php(264): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Application.php(859): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Application.php(206): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /Users/sahaj/Sites/rw84test/vendor/symfony/console/Application.php(125): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /Users/sahaj/Sites/rw84test/vendor/drush/drush/src/Runtime/Runtime.php(112): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /Users/sahaj/Sites/rw84test/vendor/drush/drush/src/Runtime/Runtime.php(41): Drush\Runtime\Runtime->doRun(Array)
#19 /Users/sahaj/Sites/rw84test/vendor/drush/drush/drush.php(64): Drush\Runtime\Runtime->run(Array)
#20 /Users/sahaj/Sites/rw84test/vendor/drush/drush/drush(4): require('/Users/sahaj/Si...')
#21 {main}
Error: Call to a member function getSource() on null in Drupal\media\Entity\Media->getSource() (line 162 of /Users/sahaj/Sites/rw84test/web/core/modules/media/src/Entity/Media.php).
 [error]  Drush command terminated abnormally due to an unrecoverable error.
sahaj at MBP in ~/Sites/rw84test/web
$ drush ms
 ----------------------------------------- ------------------------------------------------ ----------- ------- ---------- ------------- ---------------------
  Group                                     Migration ID                                     Status      Total   Imported   Unprocessed   Last Imported
 ----------------------------------------- ------------------------------------------------ ----------- ------- ---------- ------------- ---------------------
  Default (default)                         file_media                                       Importing   3632    1          3631

I am on Drupal 8.4.4. Any idea what is wrong?

Thanks for help.

Needs a re-roll now that #2908282: Throw exception for source plugins without a source_module property has landed.

#21 patch applies cleanly so setting back to Needs Review state.

@heddn is right -- this needs to take #2908282: Throw exception for source plugins without a source_module property into account, so this still needs some work. There's also the error @sahaj encountered in #29 to consider. @sahaj, can you provide steps to reproduce?

@sahaj That seems to happen when the media bundle is not found, e.g. the map may need one more entry, or the value assigned is not a valid media bundle.

@marvil07 Thanks for your reply. Actually I'm trying to migrate some Scald images into Media Entity. I have also applied this patch https://www.drupal.org/files/issues/scald-atom-migrate-source-2772129-2.... but I'm certainly missing something as their is no specific doc or instruction.

The patch in #21 doesn't apply to 8.5.1

I copied the file_media.yml from this patch to own module's config/install/migrate_plus.migration.file_media.yml

A question with regards to a Drupal 7 file to Drupal 8 media entity migration; I assume we need to perform a d7_file first and then perform the file_media template migration?

source_module seems to be missing from the source plugin annotation.

The patch in #21 needs a re-roll.

My apologies, it seems the patch works nicely but there's a bug in the cweagans/composer-patches project with patches that only add plugins:
https://github.com/cweagans/composer-patches/issues/43

I fixed my previous remark concerning the source_module & destination_module (see CR https://www.drupal.org/node/2831566).

Note: this patch is making it impossible to add new media types in 8.5.1.
See #2959869-8: Creating Media Type gives this Error: Call to a member function getName() for more (which I marked duplicate with this).
We already knew it would (see comments #30 and #33).
Just sayin'. ;)

Thanks,
-Derek

(Also note that patch #5 over there seems to mitigate the problem in my local testing, although that's probably the wrong fix overall).

Trying to pick this up.

According to #21 and #27, let's use #2930514: [meta] Migration to media as the meta and make this one only about adding source/destination plugins. I've updated the IS's of both issues to account for that.

Now, about this specific issue, some questions intended for me to get the correct context :)

+++ b/core/modules/media/src/Plugin/migrate/source/File.php
@@ -0,0 +1,132 @@
+  protected function initializeIterator() {
+    // Only query permanent files.
+    $query = $this->fileStorage
+      ->getQuery()
+      ->condition('status', FILE_STATUS_PERMANENT);

Is this only a PoC or the intention is to really migrate all permanent files on the system into media entities?
IMHO if we do that, we should at least provide a mechanism for sites to opt-out or identify some of the files that should not be migrated. I can see several use-cases where sites wouldn't necessarily want media entities wrapping some files, such as user profile pictures, webform upload fields, etc.

+++ b/core/modules/media/migration_templates/file_media.yml
@@ -0,0 +1,39 @@
+    map:
+      gif: image
+      jpg: image
+      png: image
+      jpeg: image
+      txt: file
+      doc: file
+      docx: file
+      ppt: file
+      pptx: file
+      xls: file
+      xlsx: file
+      pdf: file
+      odf: file
+      odg: file
+      odp: file
+      ods: file
+      odt: file
+      fodt: file
+      fods: file
+      fodp: file
+      fodg: file
+      key: file
+      numbers: file
+      pages: file

If we are indeed migrating all files, there should be a way of mapping also the files with extensions that don't match any of these (and without extension). (this was pointed out above already)

About this same hunk ^^ :
We are assuming the media types file and image exist on the site when this migration is executed. Is this a safe assumption to make?

Maybe we can defer this source plugin to a point where we have the upgrade path from fields ( #2930513: File/image field to media field migration ) ironed out and know exactly how the whole process is going to be?

I just did a quick review of the work here. Maybe the tests could be saved. But all that mapping in the destination should take place in the process section of a migration. And we already have a re-usable entity source in D8 (core/modules/migrate_drupal/src/Plugin/migrate/source/ContentEntity.php). So we don't need a source.

That could be taken as bad news. Its good because that means we can go strait to writing the migration! Re-titling.

We are assuming the media types file and image exist on the site when this migration is executed. Is this a safe assumption to make?

Apparently not; that's must be the reason why the test result in #38 fails with a Error: Call to a member function getSource() on null. I encountered this when trying to import files which are defined to be turned into media file; by creating this file media type I could successfully import without triggering that error!

For sites that are using file_entity but not media in Drupal 7 and want to migrate to media I created https://www.drupal.org/project/file_entity_migration.

I am experimenting now with media in Drupal 7 so I can help with a patch here. Looks like the resulting patch would look quite similar to what I did at https://www.drupal.org/project/file_entity_migration.

Here is a patch that migrates media from Drupal 7 to 8. I restricted the migration so it only migrates fields of type file that are using the Media Browser widget, like this one:

Here is how I configured the source (Drupal 7) and destination (Drupal 8) sites:

Source (Drupal 7)

Installed media module.

drush dl media
drush en media

Added a text field to the Image file type:

Added a Media Image field to the Article content type that uses Media Browser widget:

Adjusted the formatter of Media Image to render the entity:

Created an article:

Destination (Drupal 8)

composer require drupal/migrate_tools:4.x-dev drupal/migrate_upgrade:3.x-dev drupal/migrate_plus:4.x-dev
drush en -y migrate_tools,migrate_upgrade,migrate_plus,media
drush migrate:upgrade --legacy-db-key=drupal7 --legacy-root=/var/www/drupal/drupal7 --configure-only
drush -v migrate:import --execute-dependencies --tag=Configuration --group=migrate_drupal_7
drush -v migrate:import --execute-dependencies --tag=Content --group=migrate_drupal_7

Then I navigated to admin/content and found the article:

Here is the edit form:

The migration created:

• The Sample Field at the Image Media bundle.
• The Media Image field at the article content type that points to Image Media bundle entities.
• The article, containing the referenced image media entity.

Thoughts

This approach depends on three contributed modules: migrate_drupal, migrate_tools, and migrate_upgrade. How could we handle that? Should some of this go to a contributed module?

Hi
I tested all the solutions here but non of them satisfied our needs.

Especially the proposed file to media migration has huge drawbacks:
- no field creation for media entity references
- no translation possible (if an image field is translated)
- no duplicate image / file detection

Because of that, I created a custom module with an example migration path to convert file / image fields into media references and migrate all the entities in a way, that all the mentioned drawbacks are solved.

https://www.drupal.org/project/migrate_file_to_media

I hope. this helps somebody to migrate the old file / image fields to Drupal 8 Core Media entities.

I have created https://www.drupal.org/project/media_migration as a proposal to migrate media from 7 to 8. I have already used it on a couple projects so I look forward for feedback on how to improve it so it can be moved into this issue as a core patch.

Added migration logic to migrate Drupal 7's image fields into Drupal 8's image media fields at https://www.drupal.org/project/media_migration.

As a side note, I know this issue is about Media Migration, but I wanted to relay a suggestion to add error handling to Media module to mitigate this elusive error:
Error: Call to a member function getSource() on null in Drupal\media\Entity\Media->getSource()

Confirmed here:
/issues/2835825#comment-12681286

Something like "Wrong media type exception" would do. Otherwise hunting for it with a WSOD is hard, I'm updating the Unsplash module, and I've cought it there.

Okay, so...now that https://www.drupal.org/project/media_migration exists, do we still need this issue?

If we do, then probably it needs to be rescoped and redone. Possibly closed and superseded with another one. I think the thing we still need to talk about is how to migrate existing D8/D9 image fields to media.

Reroll patch created for 9.2.x.

Fixing fail test.

Fixed tests, Please have a look and advise for the next.

@vsujeetkumar, thanks for the interest in this issue but the next step here, in #58, was to decided if this was needed.

In response to #58, it would be great to have media migration is core. However, migration is also scheduled to be removed from core (eventually) so it is probably more prudent to put time and effort into working on the contrib module. There will be wiki pages to update ...

As of #71 this issue is POSTPONED on a stable release of the media_migration module.

All work and improvements should be done there. As media was never a core module in Drupal 7, there doesn't have to be Drupal 8+ core migration logic for that. That can be done in contrib "media_migration" module.

Use and improve it to migrate things from and into media, like the module page describes.

Once media_migration is stable, we can decide here to put all the code into core or set this won't fix. If a core maintainer decides to set this won't fix before, that's also fine.

Please do not post further patches here, instead help to solve media_migration issues!

I have a Drupal 9 install post Drupal 7 migration where files exist in the Files area but not in the Media area. The new site is in a good state, and I'm not sure if other Drupal 7 to 9 migrations would create duplicates or conflict with what's already in the Drupal 9 site's Files area. I also don't know if media_migration actually has anything yet for files that exist in a Drupal 9 Files area but not in Media. All I saw was stuff for Drupal 7, including patch 72 here, which is not what I needed. I basically don't care about Drupal 7 and just want Drupal 9 Files converted to Drupal 9 Media.

For anyone else needing to migrate/convert Drupal 8/9/10 Files to Media:
The patch from #38 is exactly what I needed. After patching, I copied file_media.yml to my own module's config/install/migrate_plus.migration.file_media.yml like #36 mentions and changed the definition for file to document like Drupal 9/10 has. The only other change needed was to change Unicode::strtolower to mb_strtolower in core/modules/media/src/Plugin/migrate/source/File.php. I ran drush mim file_media and it successfully added Media entries.