Migration path from Drupal 7.x

With all that said, here is a start... I needed to do a migration of a field collection to d8 from d7 as a different entity, but this source plugin was handy! I don't know if I should make this a new "feature request" or leave it here.

I'm trying to validate this plugin for a D7 -> D8 migration, but I'm not able to make it work.
Is there any example on how to define the yml file for that plugin?

@gerzenstl worked for me.
However still getting the below error whenever I run upgrade_d7_field_formatter_settings migration;

The "field_collection_view" plugin does not exist. (C:\wamp\www\dev\drupal\core\lib\Drupal\Component\Plugin\Discovery\DiscoveryTrait.php:52)[error]

Please keep in mind the plugin in the patch above will not migrate field_collection fields over. It will migrate field collection entities over. Field Collection Fields are much like entity reference fields. Note that the plugin created above is a source plugin, not a process plugin that uses the cck_field classes. That part still needs to be built... or at the very least a migration template... the above is a start that I contributed as I mapped field collection items to paragraphs.

The use of this plugin is simply

id: simple_field_collection
label: Simple Field Collection
migration_group: Field Collections
source:
  plugin: d7_field_collection_item
  field_name: field_simple_collection
destination:
  plugin: entity:field_collection_item
process:
  item_id: item_id
  revision_id: revision_id
  field_name: field_name
  field_text_1: field_text_1
  field_text_2: field_text_2

And then somewhere in your other migrations use something like

id: upgrade_d7_node_blog
migration_tags:
    - 'Drupal 7'
migration_group: migrate_drupal_7
label: 'Nodes (Blog entry)'
source:
    plugin: d7_node
    node_type: blog
process:
    nid: nid
    type: type
    langcode: [{ plugin: default_value, source: language, default_value: und }, { plugin: static_map, map: { und: en } }]
    title: title
    uid: node_uid
    status: status
    created: created
    changed: changed
    promote: promote
    sticky: sticky
    revision_uid: revision_uid
    revision_log: log
    revision_timestamp: timestamp
    body: body
    field_simple_collection: 
        #use a iteration plugin to get all the fields... for explaination sake using just the first item.
        plugin: migration
        migration: simple_field_collection
        source: field_simple_collection/0/however-the-id-is-stored-on-this-field
destination:
    plugin: 'entity:node'
migration_dependencies:
    required: [upgrade_d7_user, upgrade_d7_node_type, simple_field_collection]
    optional: [upgrade_d7_field_instance]
  

@jmuzz I've already tried the patch you've mentioned but seems it is not stable enough, since when I've used it I got a higher number of failed field migrations

@generalredneck I am not sure if this is the right way to import field collection fields, but I was able to do so by mapping it in d7_field.yml file (attached)
I will be looking to solving this;
The "field_collection_view" plugin does not exist. (C:\wamp\www\dev\drupal\core\lib\Drupal\Component\Plugin\Discovery\DiscoveryTrait.php:52)
I will try the suggested solution to migrate content.

Thank you,
Geo

Clever. That should work if the d7 configuration and d8 configuration match... that will get you the "Field" itself, but I feel you probably will still have to work on it's content. That's where my solution above comes in.

@geosalameh: your approach works fine to migrate the field collections (fields and their instances). Regarding the field_collection_view error, isn't related to the module with the same name? Because I'm don't get that error.

@generalredneck: the patch that I was testing (trying to make it work) before, was very similar to the one you shared. But it's not working, I'm still not able to migrate the content for those fields.

@generalredneck I've tried to create a migration template in modules/field_collection/migration_templates/d7_simple_field_collection.yml with the following content;

id: simple_field_collection
label: Simple Field Collection
migration_group: Field Collections
source:
  plugin: d7_field_collection_item
  field_name: field_link_group
destination:
  plugin: entity:field_collection_item
process:
  item_id: item_id
  revision_id: revision_id
  field_link: field_link
  field_text_link: field_text_link

But still not showing once I run "drush ms".
Also you've mentioned a second yaml file upgrade_d7_node_blog which usually is auto-generated by "drush migrate-upgrade" or should we create it in this case?

Thank you for you help

@gerzenstl I am not sure why I am getting this error, in addition to this;

The "text_plain" plugin does not exist. (C:\wamp\www\dev\drupal\core\lib\Drupal\Component\Plugin\Discovery\DiscoveryTrait.php:52)

Which is related to #2726803: Field formatters with names different than their field type can not be migrated

Btw, Field Collection Views is not installed on my site.

I've written a custom PHP class to migrate Field Collection data from 7 to 8, it does the following:

• Get all Field Collection Bundles from Drupal 7 site
• Query all the fields related to field collections, in order to generate a tree like array (parent - children)
• Insert Field Collection Items + their revisions
• Migrate field collection bundle tables from field_data_ (Drupal 7) to node (Drupal 8) + revision tables
• Dynamic function that loops over field collection fields, map their tables, and import their data + revision tables

Notes:
- Run this script once you've completed all migration imports (especially fields, files and nodes...)
- You need to run Clear Cache after running this script, in order for Drupal to update entity cache.

File: migrate_field_collection_data.php

<?php

class MigarteFC
{
  // Drupal Versions
  const D7 = 'D7';
  const D8 = 'D8';

  // Define DB connection
  private $host       = "";
  private $username   = "";
  private $password   = "";
  private $d7_db      = "";
  private $d8_db      = "";

  // Connection strings
  protected $conn_d7;
  protected $conn_d8;

  public function __construct()
  {
    try {
      $this->conn_d7 = new PDO("mysql:host={$this->host};dbname={$this->d7_db}", $this->username, $this->password);
      $this->conn_d8 = new PDO("mysql:host={$this->host};dbname={$this->d8_db}", $this->username, $this->password);
      // set the PDO error mode to exception
      $this->conn_d7->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
      $this->conn_d8->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

      echo "DB Connected successfully";
    } catch(PDOException $e) {
      echo "DB Connection failed: " . $e->getMessage();
    }
  }

  private function _getSchema($table, $version) {
    $schema = array();
    $schema['D7']['field_collection_item'] = array('item_id', 'revision_id', 'field_name', 'uuid');
    $schema['D7']['field_collection_item_revision'] = array('revision_id', 'item_id');

    $schema['D8']['field_collection_item'] = array('item_id', 'revision_id', 'field_name', 'uuid', 'host_type');
    $schema['D8']['field_collection_item_revision'] = array('item_id', 'revision_id');

    return implode( $schema[$version][$table] , ', ');
  }

  public function getFcBundles() {
    $stmt = $this->conn_d7->prepare("
      SELECT field_name FROM field_config
      WHERE module = 'field_collection'
    ");
    $stmt->execute();

    $result = $stmt->fetchAll(PDO::FETCH_ASSOC);
    return array_column($result, 'field_name');
  }

  public function getFcBundleFields($bundle) {
    $stmt = $this->conn_d7->prepare("
      SELECT field_name FROM field_config_instance
      WHERE bundle = '$bundle'
    ");
    $stmt->execute();

    $result = $stmt->fetchAll(PDO::FETCH_ASSOC);
    return array_column($result, 'field_name');
  }

  public function mapFieldCollectionItemTable($bundle) {
    $table = 'field_collection_item';
    $table_fields = $this->_getSchema($table, MigarteFC::D7);

    $stmt = $this->conn_d7->prepare("
      SELECT $table_fields FROM $table
      WHERE field_name = '$bundle'
    ");

    $stmt->execute();
    $result = $stmt->fetchAll(PDO::FETCH_ASSOC);

    $i = 0;
    $table_fields = $this->_getSchema($table, MigarteFC::D8);
    foreach ($result as $row) {
      try {
        $table_values = "'" . implode($row, "', '") . "', 'node'";

        $sql = "INSERT INTO $table ($table_fields) VALUES ($table_values)";
        $this->conn_d8->exec($sql);
        echo ">>>> Query #" . ++$i . ": Inserted" . PHP_EOL;
      } catch (PDOException $e) {
        echo $sql . PHP_EOL . $e->getMessage();
        echo "\n= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n";
      }
    }
  }

  public function mapFieldCollectionItemRevisionTable($bundle) {
    $table = 'field_collection_item_revision';
    $table_fields = $this->_getSchema($table, MigarteFC::D7);

    $stmt = $this->conn_d7->prepare("
      SELECT $table.* FROM $table
      INNER JOIN field_collection_item
      ON field_collection_item.item_id = field_collection_item_revision.item_id
      WHERE field_collection_item.field_name = '$bundle'
    ");

    $stmt->execute();
    $result = $stmt->fetchAll(PDO::FETCH_ASSOC);

    $i = 0;
    $table_fields = $this->_getSchema($table, MigarteFC::D8);
    foreach ($result as $row) {
      try {
        $table_values = "'" . implode($row, "', '") . "'";

        $sql = "INSERT INTO $table ($table_fields) VALUES ($table_values)";
        $this->conn_d8->exec($sql);
        echo ">>>> Query #" . ++$i . ": Inserted" . PHP_EOL;
      } catch (PDOException $e) {
        echo $sql . PHP_EOL . $e->getMessage();
        echo "\n= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n";
      }
    }
  }

  public function mapFieldCollectionBundleTable($bundle) {
    // D7 Table
    $d7_field_table_name          = 'field_data_'.$bundle;
    $d7_field_revision_table_name = 'field_revision_'.$bundle;

    // D8 Table
    $d8_field_table_name          = 'node__'.$bundle;
    $d8_field_revision_table_name = 'node_revision__'.$bundle;

    // Get field data from D7
    $stmt = $this->conn_d7->prepare("
      SELECT * FROM $d7_field_table_name
    ");
    $stmt->execute();

    $result = $stmt->fetchAll(PDO::FETCH_ASSOC);

    // Migrate fetched data to D8 tables
    $i = 0;
    foreach ($result as $row) {
      ++$i;
      // we're forcing language to English,
      // the default value 'und' wasn't loading the data in Drupal edit node
      $row['langcode'] = ('und' == $row['language']) ? 'en' : $row['language'];
      // $row['langcode'] = $row['language'];
      unset($row['language']);
      unset($row['entity_type']);

      // Handle field value case
      if (isset($row[$bundle.'_value'])) {
        $row[$bundle.'_target_id'] = $row[$bundle.'_value'];
        unset($row[$bundle.'_value']);
      }

      $table_fields = implode(array_keys($row), ", ");
      $table_values = '';
      foreach ($row as $value) {
        $table_values.= "'" . str_replace("'", "\'", $value) . "', ";
      }
      $table_values = rtrim($table_values, ', ');

      try {
        $sql = "INSERT INTO $d8_field_table_name ($table_fields) VALUES ($table_values)";

        $this->conn_d8->exec($sql);
        echo ">>>> Query #" . $i . ": Inserted" . PHP_EOL;
      } catch (PDOException $e) {
        echo $sql . PHP_EOL . $e->getMessage();
        echo "\n= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n";
      }

      try {
        $sql = "INSERT INTO $d8_field_revision_table_name ($table_fields) VALUES ($table_values)";

        $this->conn_d8->exec($sql);
        echo ">>>> Query Revision #" . $i . ": Inserted" . PHP_EOL;
      } catch (PDOException $e) {
        echo $sql . PHP_EOL . $e->getMessage();
        echo "\n= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n";
      }
    }
  }

  public function mapFieldCollectionFieldsTables($bundle) {
    // Get bundle fields
    $fc_fields = $this->getFcBundleFields($bundle);

    echo "\n____________________________________________________________\n";
    print_r($fc_fields);
    echo "\n____________________________________________________________\n";

    $i = 0;
    foreach ($fc_fields as $fc_field) {
      // D7 Table
      $d7_field_table_name          = 'field_data_'.$fc_field;
      $d7_field_revision_table_name = 'field_revision_'.$fc_field;

      // D8 Table
      $d8_field_table_name          = 'field_collection_item__'.$fc_field;
      $d8_field_revision_table_name = 'field_collection_item_revision__'.$fc_field;

      // Get field data from D7
      $stmt = $this->conn_d7->prepare("
        SELECT * FROM $d7_field_table_name
        WHERE bundle = '$bundle'
      ");
      $stmt->execute();

      $result = $stmt->fetchAll(PDO::FETCH_ASSOC);

      // Migrate fetched data to D8 tables
      foreach ($result as $row) {
        ++$i;
        $row['langcode'] = $row['language'];
        unset($row['language']);
        unset($row['entity_type']);

        // Handle image field case
        if (isset($row['field_image_fid'])) {
          $row['field_image_target_id'] = $row['field_image_fid'];
          unset($row['field_image_fid']);
        }

        $table_fields = implode(array_keys($row), ", ");
        $table_values = '';
        foreach ($row as $value) {
          $table_values.= "'" . str_replace("'", "\'", $value) . "', ";
        }
        $table_values = rtrim($table_values, ', ');

        try {
          $sql = "INSERT INTO $d8_field_table_name ($table_fields) VALUES ($table_values)";

          $this->conn_d8->exec($sql);
          echo ">>>> Query #" . $i . ": Inserted" . PHP_EOL;
        } catch (PDOException $e) {
          echo $sql . PHP_EOL . $e->getMessage();
          echo "\n= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n";
        }

        try {
          $sql = "INSERT INTO $d8_field_revision_table_name ($table_fields) VALUES ($table_values)";

          $this->conn_d8->exec($sql);
          echo ">>>> Query Revision #" . $i . ": Inserted" . PHP_EOL;
        } catch (PDOException $e) {
          echo $sql . PHP_EOL . $e->getMessage();
          echo "\n= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n";
        }
      }
    }
  }
}

// Init
$migrate_fc = new MigarteFC();

// 1 - Get all Field Collection bundles
$fc_bundles = $migrate_fc->getFcBundles();

// 2 - Loop over each bundle and it's fields
foreach ($fc_bundles as $fc_bundle) {
  echo PHP_EOL . "______________________________[ $fc_bundle ]______________________________\n";
  // 3a - Migrate Field Collection bundle/group
  $migrate_fc->mapFieldCollectionBundleTable($fc_bundle);

  // 3a - Migrate field_collection_item
  echo PHP_EOL . "______________________________[ field_collection_item ]______________________________\n";
  $migrate_fc->mapFieldCollectionItemTable($fc_bundle);

  // 3b - Migrate field_collection_item_revision
  echo PHP_EOL . "______________________________[ field_collection_item_revision ]______________________________\n";
  $migrate_fc->mapFieldCollectionItemRevisionTable($fc_bundle);

  // 3c - Migrate Field Collection Fields data & revisions
  $migrate_fc->mapFieldCollectionFieldsTables($fc_bundle);

  echo "____________________________________________________________\n";
}

Feel free to modify this script

Btw, I've tried to write a module for this but didn't have enough time, so I went with the easiest option ;)

geosalameh, how to run and where to put your script.

Hello dhruva2,

1. Install UUID module in Drupal 7 site
2. Modify drupal/core/modules/field/migration_templates/d7_field.yml (attached in comment #18) in order to include "field_collection" field types
3. Save the script (comment #23) to a new file migrate_field_collection_data.php and upload it to your Drupal site (In my case, I've uploaded it to Field Collection module folder (drupal/modules/field_collection/migrate_field_collection_data.php)
4. Run this script once you've completed all migration imports (especially fields, files and nodes...)
5. I ran the script using cmd line from terminal
   php drupal/modules/field_collection/migrate_field_collection_data.php (the full path may be different on your server)
6. run Clear Cache after running this script, in order for Drupal to update entity cache.

Btw this script is not taking into consideration multilingual fields, or field collection in another field collection!

Hi Geosalameh,

Thanks for updating the article. I will update it with the results.

Thanks and regards

Hi,
Do I need to use Drush to run this command.

Thanks and Regards

After the patch, use as below

id: d7_field_collection[MACHINE NAME]
migration_tags:
  - 'Drupal 7'
label: 'Field Collection [NAME]'
source:
  plugin: d7_field_collection_item
  field_name: [FIELD COLLECTION MACHINE NAME]
process:
  item_id: item_id
  revision_id: revision_id
  field_name: field_name
  host_type: entity_type
  host_entity_id: entity_id
destination:
  plugin: 'entity:field_collection_item'
migration_dependencies:
  required: { }

Hi heshanlk,
About which patch you are talking about? d7field.yml patch?

Ho sorry for the confusion, the one attached to the comment #31

Hi Heshankl,

So I need to add https://www.drupal.org/files/issues/2757989-patch-migrate-multi-value-fi... to
core/modules/field/migration_templates

and than

where to add this code in which file, sorry for asking lots of questions as I am naive

id: d7_field_collection[MACHINE NAME]
migration_tags:
- 'Drupal 7'
label: 'Field Collection [NAME]'
source:
plugin: d7_field_collection_item
field_name: [FIELD COLLECTION MACHINE NAME]
process:
item_id: item_id
revision_id: revision_id
field_name: field_name
host_type: entity_type
host_entity_id: entity_id
destination:
plugin: 'entity:field_collection_item'
migration_dependencies:
required: { }

Thanks and regards

You need to apply the patch, read here https://www.drupal.org/patch/apply. And then you can create a yml files for the migrations of each field collection you have. I know I'm not much help here, you need to read how to create yml files for the migrations etc.

also, do I need to change machine name for each FC.yml

Guys after databases migration I have the following error. I am unable to understand what is the reason

TypeError: Argument 1 passed to Drupal\Core\Entity\Entity\EntityFormDisplay::buildForm() must implement interface Drupal\Core\Entity\FieldableEntityInterface, null given, called in /home/dhruva/webapps/d82/modules/field_collection/src/Plugin/Field/FieldWidget/FieldCollectionEmbedWidget.php on line 64 in Drupal\Core\Entity\Entity\EntityFormDisplay->buildForm() (line 159 of core/lib/Drupal/Core/Entity/Entity/EntityFormDisplay.php).

Hi geosalameh,
Thank you for the superb script, it did exactly what you said it will do. I only have one problem though. After clearing cache nothing seem to happen. The parent node is not attached to the field collection. Am I missing something. Is there any chance you can share your node migration script (node.yml). Most especially node to field collection mapping.
Thank you once again.

Hi geosalameh, Your script in #23 was helpful.

Thanks!

Hi

Tried the patch in #31 and I got the following error

TypeError: Argument 1 passed to _field_collection_field_item_list_full() must implement interface Drupal\Core\Field\FieldItemListInterface, null given, called in                              [error]
/var/www/html/docroot/modules/contrib/field_collection/src/Entity/FieldCollectionItem.php on line 311 in _field_collection_field_item_list_full() (line 145 of
/var/www/html/docroot/modules/contrib/field_collection/field_collection.module) #0 /var/www/html/docroot/modules/contrib/field_collection/src/Entity/FieldCollectionItem.php(311):
_field_collection_field_item_list_full(NULL)
#1 /var/www/html/docroot/modules/contrib/field_collection/src/Plugin/migrate/destination/EntityFieldCollection.php(28):
Drupal\field_collection\Entity\FieldCollectionItem->setHostEntity(Object(Drupal\node\Entity\Node))
#2 /var/www/html/docroot/core/modules/migrate/src/MigrateExecutable.php(224): Drupal\field_collection\Plugin\migrate\destination\EntityFieldCollection->import(Object(Drupal\migrate\Row),
Array)

Any ideas

#31 worked for me, except I had to change the annotation from source_provider to source_module

I had to do the same as #41 to get drush to clear the cache even after I installed the patch.