Create Media types/bundles based upon available File/Media types

By "file type" do you mean the type column that the file_entity module added to the file_managed table?

#3159580: Migrate media entities from sources that aren't using Media or File Entity module introduced a plugin system with media type migration support.

The attached patch does the same for File Entity → Media migrations.

I also added media source field widget and media field formatter migrations.

Same patch as in #5, without extra trailing new line errors.

1. Manually tested, works great! 🚢

2. +++ b/migrations/d7_file_entity.yml
   @@ -17,7 +17,7 @@ process:
   -  bundle: type
   +  bundle: bundle
   

   🤔 Was this a bug in HEAD?

3. +++ b/migrations/d7_file_entity.yml
   @@ -27,6 +27,5 @@ destination:
   -    - d7_field_instance
   

   🤔 I think this can be removed because file entities are not actually fieldable?

4. +++ b/migrations/d7_file_entity_source_field_config.yml
   @@ -1,22 +1,20 @@
   -id: d7_media_source_field_config
   +id: d7_file_entity_source_field_config
   

   👍 This rename makes it clearer that this migration is specific to file_entity source data.

   🤔 But "source field" is a D8 Media-specific concept. So perhaps d7_file_entity_to_media_source_field_config would be clearer?

   (Similar observation for the other migration .yml files touched by this patch by the way.)

5. +++ b/src/Annotation/FileEntityDealer.php
   @@ -0,0 +1,65 @@
   +   * The id of the destination media type's source plugin.
   

   🤓 Nit: s/id/ID/

6. +++ b/src/EventSubscriber/MediaMigrationSubscriber.php
   @@ -72,22 +72,6 @@ class MediaMigrationSubscriber implements EventSubscriberInterface {
   -    // Skip migrating field instances whose destination bundle does not exist.
   -    if (in_array($source->getPluginId(), [
   -      'd7_field_instance',
   -      'd7_field_instance_per_view_mode',
   -      'd7_field_instance_per_form_display',
   -    ])) {
   -      if ($row->getSourceProperty('entity_type') == 'file') {
   -        // Don't migrate bundles which don't exist in the destination.
   -        $media_bundle = $row->getSourceProperty('bundle');
   -        if (!$this->entityTypeManager->getStorage('media_type')->load($media_bundle)) {
   -          $field_name = $row->getSourceProperty('field_name');
   -          throw new MigrateSkipRowException('Skipping field ' . $field_name . ' as its target is ' . $media_bundle . ', which does not exist in the destination.');
   -        }
   -      }
   -    }
   -
   

   🤔🤩 All of this logic is obsolete because it's being replaced with actual migration definitions, right?

7. +++ b/src/FileEntityDealerManager.php
   @@ -0,0 +1,108 @@
   +   * Constructs a new \Drupal\Core\Block\BlockManager object.
   

   🤓 C/P remnant

8. +++ b/src/FileEntityDealerManager.php
   @@ -0,0 +1,108 @@
   +      'Drupal\media_migration\FileEntityDealerPluginInterface',
   +      'Drupal\media_migration\Annotation\FileEntityDealer'
   

   🤔 Why not use FQCN::class here?

9. +++ b/src/FileEntityDealerManager.php
   @@ -0,0 +1,108 @@
   +  protected function getDefinitionsByTypeAndScheme($type, $scheme) {
   

   🤔 Isn't this identical to \Drupal\media_migration\FileDealerManager::getDefinitionsByFieldTypeAndScheme()?

   Then why not make FileEntityDealerManager extends FileDealerManager and overriding the constructor?

   Never mind, it's subtly different! For file the parameters are string $scheme, string $mime, here (for file_entity) they are $type, $scheme. Makes sense.

10. +++ b/src/FileEntityDealerManagerInterface.php
    @@ -0,0 +1,26 @@
    +  public function createInstanceFromTypeAndScheme($type, $scheme);
    

    🤔 Why not

    public function createInstanceFromTypeAndScheme(string $type, string $scheme) : FileEntityDealerPluginInterface;
    

    ?

11. +++ b/src/MigratePluginAlterer.php
    @@ -129,6 +153,28 @@ class MigratePluginAlterer {
    +        // entities require their host entity.
    

    🤓 Übernit: s/host entity/host entity type/

12. +++ b/src/Plugin/media_migration/file_entity/FileBase.php
    @@ -0,0 +1,103 @@
    + * Abstract plugin class for file-based media migration source plugins.
    

    🤓 s/file-based/file entity-based/

13. +++ b/src/Plugin/media_migration/file_entity/Youtube.php
    @@ -0,0 +1,17 @@
    + *   schemes = {"youtube"},
    

    So #3116634: Migrate Drupal 7 remote videos to Drupal 8|9 remote video media entities will not be necessary anymore, because this supports remote video stored as streams (those stream wrappers are provided by https://www.drupal.org/project/media_youtube and https://www.drupal.org/project/media_vimeo), which Media in Drupal 7 stores in file_entity entities under the hood.

    So, can you confirm that this patch makes #3116634 obsolete?

    If so — wow, this solution is much more elegant than #3116634! The power of iteration and learning! 🤩

14. +++ b/src/Plugin/migrate/process/MediaInternetFieldValue.php
    @@ -0,0 +1,40 @@
    +class MediaInternetFieldValue extends ProcessPluginBase {
    +
    +  /**
    +   * {@inheritdoc}
    +   */
    +  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    +    // We're operating with the source values.
    +    $source_values = $row->getSource();
    +    $replaced = preg_replace([
    +      '/^youtube:\/\/v\//i',
    +      '/^vimeo:\/\/v\//i',
    +    ], [
    +      'https://www.youtube.com/watch?v=',
    +      'https://vimeo.com/',
    +    ], $source_values['uri']);
    +
    ...
    +      return $replaced;
    +    }
    +
    +    return $value;
    

    Ah, we still have this, which is very similar to what #3116634: Migrate Drupal 7 remote videos to Drupal 8|9 remote video media entities contained.

    The code I praised in my previous comment is for migrating the actual media into the media library, from a file entity.

    This code on the other hand is for migrating a reference to that media, and ensures it ends up finding (and referencing) the appropriate media ID.

    👍

15. +++ b/src/Plugin/migrate/source/d7/FileEntityFieldFormatter.php
    @@ -0,0 +1,94 @@
    +    // Media Migration wants to hide "created", "name", "thumbnail" and "uid"
    +    // base fields for the default view mode.
    +    $iterator = parent::initializeIterator();
    +    $rows = [];
    +    foreach ($iterator->getArrayCopy() as $item) {
    +      [
    +        'source_field_name' => $source_field_name,
    +      ] = $item;
    +
    +      $field_names = [
    +        $source_field_name => FALSE,
    +        'created' => TRUE,
    +        'name' => TRUE,
    +        'thumbnail' => TRUE,
    +        'uid' => TRUE,
    +      ];
    

    🙏

    // @see \Drupal\media\Entity\Media
    

16. +++ b/src/Plugin/migrate/source/d7/FileEntityItem.php
    @@ -19,85 +24,125 @@ use Drupal\migrate_drupal\Plugin\migrate\source\d7\FieldableEntity;
    +   * Constructs a FileEntityType instance.
    

    🤓 s/Type/Item/

Re #10:

1. 🤩
2. No, but it was a kind of limitation (type is/was the bundle of the file entity, and not the destination media type ID.)
3. File entities are fieldable, but they do not always have extra fields. So this dependency added dynamically, only when a file entity bundle HAS a field.
   

   +++ b/src/Plugin/migrate/D7FileEntityDeriver.php
   @@ -0,0 +1,163 @@
   +            // Add bundle field processes.
   +            $migration = \Drupal::service('plugin.manager.migration')->createStubMigration($derivative_definition);
   +            $process_keys_before = array_keys($derivative_definition['process']);
   +            assert($migration instanceof Migration);
   +            $this->fieldDiscovery->addBundleFieldProcesses($migration, 'file', $type);
   +            $derivative_definition = $migration->getPluginDefinition();
   +            if (!empty(array_diff(array_keys($derivative_definition['process']), $process_keys_before))) {
   +              $derivative_definition['migration_dependencies']['required'][] = 'd7_field_instance';
   +            }
   

   +++ b/src/MigratePluginAlterer.php
   @@ -111,6 +111,30 @@ class MigratePluginAlterer {
   +    // Collect all file entity -> media entity migration which have additional
   +    // fields. Although these migrations depend on the "d7_field_instance"
   +    // migration (and implicitly on "d7_field" migration), it isn't enough:
   +    // the parent media type entity has to exist prior its all field storage
   +    // entity being created, so "d7_field" (and maybe "d7_field_instance") has
   +    // to depend on the media type migrations which have additional fields.
   +    $media_types_with_fields_migration_ids = array_reduce($migrations, function (array $carry, array $migration_def) {
   +      $deps_required = isset($migration_def['migration_dependencies']['required'])
   +        ? $migration_def['migration_dependencies']['required']
   +        : [];
   +
   +      if (!in_array('d7_field_instance', $deps_required, TRUE)) {
   +        return $carry;
   +      }
   +
   +      foreach ($deps_required as $requirement) {
   +        $requirement_parts = explode(':', $requirement);
   +        if ($requirement_parts[0] === 'd7_file_entity_type') {
   +          $carry[$requirement] = $requirement;
   +        }
   +      }
   +      return $carry;
   +    }, []);
   +
        foreach ($migrations as &$migration) {
          $migration_stub = $this->pluginManagerMigration->createStubMigration($migration);
          $source = NULL;
   @@ -129,6 +153,28 @@ class MigratePluginAlterer {
   
   @@ -129,6 +153,28 @@ class MigratePluginAlterer {
            if (is_a($source, ViewMode::class)) {
              $this->mapFileToMediaBundle($migration);
            }
   +
   +        // D7 field storage and field instance migrations should depend on
   +        // media types which have extra fields, because field storage config
   +        // entities require their host entity.
   +        $field_storage_and_instance_source_plugin_ids = [
   +          'd7_field',
   +          'd7_field_instance',
   +        ];
   +        if (
   +          in_array($migration['source']['plugin'], $field_storage_and_instance_source_plugin_ids) &&
   +          !empty($media_types_with_fields_migration_ids)
   +        ) {
   +          $required_migration_deps = isset($migration['migration_dependencies']['required'])
   +            ? $migration['migration_dependencies']['required']
   +            : [];
   +          $migration['migration_dependencies']['required'] = array_unique(
   +            array_merge(
   +              $required_migration_deps,
   +              array_values($media_types_with_fields_migration_ids)
   +            )
   +          );
   +        }
   

4. Yeah, but I don't want to risk running into #2845340: migrate mapping & messages table names are truncated, can lead to incorrect mapping lookups.
5. I'll fix this ASAP.
6. Yes, exactly!
7. Again... 😄
8. 🤓
9. 🤓
10. 👍
11. 👍
12. 👍
13. Yes, I can confirm. (I also had to copy all of the test helper modules added there.)
14. Well, this is what actually transforms the original value to the new media' source field.
15. I guess I should add this comment there. Am I right?
16. 🤓

3. Ohh! 👍👍👍
4. LOL! Okay, let's keep it as-is :)

15. Yes please 🤓

Addressing everything from #11.

Interdiff does not contain documentation (README) update.

+++ b/src/Plugin/media_migration/file_entity/FileBase.php
@@ -8,6 +8,12 @@ use Drupal\migrate\Row;
+ * Not every file entity will be migrated into a media file field: for example,
+ * remote ("YouTube" or "Vimeo") file entities are migrated into a media source
+ * field with type "string". This is a base plugin class for those file entity
+ * dealer plugins which needs to migrate file entities to media entities with
+ * file-based source field (linke "file" or "image").

This matches what core/profiles/standard/config/optional/field.field.media.remote_video.field_media_oembed_video.yml does, so 👍

Follow-up at #3198938: Harden \Drupal\media_migration\MigratePluginAlterer::alterFieldMigrations(): dependency on d7_file_entity_type is not always added when necessary to d7_field_instance.