Problem/Motivation

We need a method to upgrade/migrate from D7 field collections to D8 paragraphs.

Proposed resolution

Remaining tasks

  1. field collection type => paragraphs type migration, look at d7_node_type as a pattern
  2. a field collections source entity plugin, look at node source
  3. a field collections field plugin, look at entity reference
  4. field collections => paragraphs derriver and base migration, look at node as a pattern

User interface changes

API changes

Data model changes

Original

I've created a process plugin that allows for a mostly easy migration map for field_collection in Drupal 7 to paragraphs in Drupal 8. It was used successfully on a project recently, but it still needs more work.

I will be uploading a patch for the class itself, an example class, and some documentation shortly.

CommentFileSizeAuthor
#86 interdiff.2897021.80-86.txt4.16 KBmikelutz
#86 2897021-86.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch77.85 KBmikelutz
#81 interdiff.2897021.79-81.txt2.19 KBmikelutz
#81 2897021-81.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch78.17 KBmikelutz
#79 interdiff.2897021.62-79.txt48.78 KBmikelutz
#79 2897021-79.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch78.09 KBmikelutz
#62 2897021-62.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch70.52 KBmikelutz
#40 2897021-40.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch70.6 KBmikelutz
#26 interdiff.2897021.21-26(no fixture).txt8.78 KBmikelutz
#26 2897021-26.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs(nofixture).txt65.71 KBmikelutz
#26 interdiff.2897021.21-26.txt126.73 KBmikelutz
#26 2897021-26.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch2.77 MBmikelutz
#21 2897021-21.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs(no-fixture).txt62.78 KBmikelutz
#21 2897021-21.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch2.64 MBmikelutz
#10 paragraphs-n2897021-10.interdiff.txt1.77 KBDamienMcKenna
#10 paragraphs-n2897021-10.patch10.03 KBDamienMcKenna
#2 2897021-migrate-support-for-field-collections-2.patch10.03 KBcruno
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

cruno created an issue. See original summary.

cruno’s picture

cruno’s picture

Status: Active » Needs review
DamienMcKenna’s picture

Issue summary: View changes
heddn’s picture

Status: Needs review » Needs work
  1. +++ b/README.txt
    @@ -44,3 +44,37 @@ of the paragraph itself into multilingual mode. This would raise complexity
    + * Implement the __construct() method to set the source database key.  The key
    + should match that of the migration source database from the $databases array in
    + settings.php.
    

    This isn't necessary. Assign the migration to a group that has the db key stored in it.

  2. +++ b/src/Plugin/migrate/process/MigrateD7FCtoD8Paragraph.php
    @@ -0,0 +1,204 @@
    +abstract class MigrateD7FCtoD8Paragraph extends ProcessPluginBase {
    

    This should be a field plugin, not a process plugin. Look at date or link field plugins in 8.4.x

heddn’s picture

This is only part of the solution we also need a source deriver too.

heddn’s picture

Title: Migrate support for importing field collections as paragraphs » [META] Migrate support for importing field collections as paragraphs
Category: Feature request » Plan
Issue summary: View changes

I worked out a plan with @Adita yesterday on what is needed here.

  • field collection type => paragraphs type migration, look at node as a pattern
  • a field collections source entity plugin, look at node source
  • a field collections field plugin, look at entity reference
  • field collections => paragraphs derriver and base migration, look at node as a pattern
heddn’s picture

Child issues opened.

elgordogrande’s picture

Looks like there's a namespace typo in the class: "paragraphss"

DamienMcKenna’s picture

Status: Needs work » Needs review
FileSize
10.03 KB
1.77 KB

Fixed the typoo.

colan’s picture

Status: Needs review » Needs work
+++ b/src/Plugin/migrate/process/MigrateD7FCtoD8Paragraph.php
@@ -14,7 +14,7 @@
+ * @see \Drupal\paragraphs\Plugin\migrate\process\D7FCtoD8ParagraphExample for uesage.

You got that one, but "usage" is still spelled wrong. ;)

docans’s picture

So i applied the patch to my paragraph module manually because if you try to apply the patch using git apply -v path_name.patch , it does not work.

I run the drush migrate-upgrade --legacy-db-url=mysql://user:password@server/db --legacy-root=http://example.com --configure-only so as to generate the migration script but i didnt see anything related to field collection migration.

Any idea what i did wrong here. Also what are the steps i am suppose to follow in order to successfuly migrate field collection in Drupal 7 to Paragraphs in Drupal 8 after applying the patch paragraphs-n2897021-10.patch

Thanks.

docans’s picture

Priority: Normal » Critical
colan’s picture

Priority: Critical » Normal

This is definitely not critical as it's possible to use Paragraphs if you're not migrating D7 Field Collections. Critical means that the module doesn't work at all, under any conditions.

docans’s picture

I just want to know what i need to do after applying the patch so i can successfully migrate the field collection over. I dont see any documentation on what do do after applying the patch. Can anyone please point me in the right direction. Thanks

rozh’s picture

+1 on documentation. Could you provide some hints on how to perform migration.

mikelutz’s picture

I'm working on this, and currently running into issues with multilingual setups. In particular, since field collection supported translation on the reference field and paragraphs doesn't, I can't properly map the content from one to another. I currently have a need to develop a means of migrating field collections from a multilingual d7 site over to d8 paragraphs, Which looks like it's going to depend on the patches in 2461695 to enable asymmetrical translations for paragraphs. I'll be posting my work over the next while, but I'm curious if anyone has any other thoughts on the issue. I will need to build something to meet my needs, but as far as a mergeable migration, it seems like we either need to not support multilingual field_collections with unlimited deltas or wait until asymmetrical translations for paragraphs are merged.

colan’s picture

I think it's safe to assume #2461695: Support asymmetric translations will be getting in soon as it's green/RTBC, but we should state that it's a dependency here for i18n (until it gets merged). So let's get support in for that even though there's a small chance it might change somewhat.

The other option is to support it in a follow-up, but that feels like too many moving parts here.

heddn’s picture

For anyone stumbling on this issue, the patches earlier in this issue are good interim steps to run a field collection to paragraphs migration. But they probably won't get committed. The actual work for all this is happening down in the child issues. Therefore, hiding all files so as to avoid some confusion.

miro_dietiker’s picture

@mikelutz Good to see progress with shipping the migration out of the box. I understand that this is a blocker then for field collection replacement. Let's cover this requirement in the experimental widget. I don't want to touch the classic widget functionality.
I'll redirect all efforts for this to happen into #2904705: Support asymmetric translation in experimental widget as a standalone issue.

mikelutz’s picture

This is the combined patch from all the child issues. It is passing tests, and seems to be working with the exception of translations.

mikelutz’s picture

Status: Needs work » Needs review
miro_dietiker’s picture

Can you please provide some more information about how committing of these would work?
Is the combined patch just for a combined testing and we keep committing the issues individually with individual test coverage?
Or do we need to commit the combined version?

mikelutz’s picture

@miro_dietiker - It really depends on your preference. I have done development in the individual issues because they were there, and it did break the work down into bite-sized chunks that are easy to review. In truth, they have been a pain for me, as each issue depends on the previous one to work. I have to keep rebasing my local copies to try and keep all the patches straight, and while I could do some individual unit tests on each piece, full test coverage really requires them all to be combined. It's about to get worse when I tackle multi-lingual support today, which will touch all the issues at some level. Finally, any real world testing requires all the patches together, applied in the right order, and I figure no one will actually do that, which is why I posted the combined patch here, for anyone who actually can try to use it to migrate field collections to test.

My preference as a developer would be to shut the child issues down, and continue to work on D7 Field Collection and D7 Paragraphs migration in this thread off of the latest patch I've supplied, and commit one big patch when everything is ready. The only thing that might be helpful to separate out, if it could be committed quickly, would be the changes I've made to the fixture to support more advanced testing of things like revision, widget settings, and formatter settings. That would reduce the size and increase the readability of this patch and make things easier. If that's acceptable, I can make a separate issue for it, and add a patch for the fixture which passes the current tests, and then rebase this patch against it, and we can continue work in here.

Otherwise, I'll adjust my workflow to whatever you prefer. My main goal is to get the code available for people to use and test, and ultimatly get committed in whatever way is easiest.

miro_dietiker’s picture

I'm only looking at the global test pass result while trusting on the review of migrate experts like heddn and Berdir.

It's still great to have the individual issues updated for specific reviews of the plugins for feedback.
In the best case they just confirm then that things are fine. We can always deal with improvements in follow-ups.

mikelutz’s picture

heddn’s picture

Status: Needs review » Needs work

Hmm, I tried. I really did. But after getting to half-way, I really think we need to take things slower and step-by-step. The attached patch that combines all the things is really hard to review. Let's split things out to the sub issues again. And if that means we loose some momentum, so be it.

Below is the first part of my review. When addressing, please move the changes back into the separate issue again. And let's perhaps pend the follow-on issues until we get the first issue to land. That will mean less churn and re-work.

  1. +++ b/migrations/d7_field_collection.yml
    @@ -0,0 +1,19 @@
    +id: d7_field_collection
    +label: Field Collections
    

    Can this label be more descriptive? Or perhaps make it singular to match the id?

  2. +++ b/migrations/d7_field_collection.yml
    @@ -0,0 +1,19 @@
    +
    

    Extra whitespace isn't needed.

  3. +++ b/migrations/d7_field_collection_revisions.yml
    @@ -0,0 +1,26 @@
    \ No newline at end of file
    

    Nit.

  4. +++ b/migrations/d7_paragraphs_revisions.yml
    @@ -0,0 +1,24 @@
    +      plugin: extract
    +      index:
    +        -
    +          0
    

    I think we want to retrieve the 0 and 1 position for paragraphs?

  5. +++ b/paragraphs.module
    @@ -441,3 +446,107 @@ function paragraphs_library_info_alter(&$libraries, $extension) {
    +      if ((is_a($migration['class'], FieldMigration::class, TRUE))) {
    ...
    +        if (is_a($source, FieldInstance::class)) {
    ...
    +      if (is_a($source, ViewMode::class)) {
    

    There's some work going on over in #2944597: Make the tests for plugins in plugins alter stricter that makes me think this should be a stricter check.

  6. +++ b/paragraphs.module
    @@ -441,3 +446,107 @@ function paragraphs_library_info_alter(&$libraries, $extension) {
    +      'plugin' => 'paragraphs_strip_prefix',
    

    Now all of this is in a single issue, can we consider renaming this again? 'field_collection_strip_prefix' seems much more descriptive of what we are doing.

  7. +++ b/src/Plugin/migrate/D7FieldCollectionItemDeriver.php
    @@ -0,0 +1,178 @@
    +    catch (RequirementsException $e) {
    +      return $this->derivatives;
    +    }
    ...
    +    return $this->derivatives;
    
    +++ b/src/Plugin/migrate/D7ParagraphsItemDeriver.php
    @@ -0,0 +1,164 @@
    +  public function getDerivativeDefinitions($base_plugin_definition) {
    ...
    +    catch (RequirementsException $e) {
    +      return $this->derivatives;
    +    }
    ...
    +    return $this->derivatives;
    

    Let's call parent::getDerivativeDefinitions(). That will do the same thing, but also let the parent adjust things. I'm thinking of the deriver alter logic that will hopefully land someday soon.

  8. +++ b/src/Plugin/migrate/D7FieldCollectionItemDeriver.php
    @@ -0,0 +1,178 @@
    +
    +    }
    
    +++ b/src/Plugin/migrate/D7ParagraphsItemDeriver.php
    @@ -0,0 +1,164 @@
    +    catch (RequirementsException $e) {
    +
    +    }
    

    Is this what core is doing too? Can we add a comment why we are catching and not doing anything w/ the exception?

  9. +++ b/src/Plugin/migrate/D7FieldCollectionItemDeriver.php
    @@ -0,0 +1,178 @@
    +      foreach ($types as $row) {
    +        /** @var \Drupal\migrate\Row $row */
    +        $values = $base_plugin_definition;
    +        $fc_bundle = $row->getSourceProperty('field_name');
    +        $p_bundle = $row->getSourceProperty('bundle');
    

    This loop could use some comments on what is going on.

  10. +++ b/src/Plugin/migrate/D7ParagraphsItemDeriver.php
    @@ -0,0 +1,164 @@
    +      foreach ($types as $row) {
    ...
    +          foreach ($fields[$bundle] as $field_name => $info) {
    

    This looping logic could use some descriptions on what it is doing.

  11. +++ b/src/Plugin/migrate/destination/Paragraphs.php
    @@ -0,0 +1,75 @@
    +class Paragraphs extends EntityReferenceRevisions {
    

    I think this needs to be pulled out of here into a separate issue. I don't think this will ever land in paragraphs. The logical place is over in ERR.

  12. +++ b/src/Plugin/migrate/field/FieldCollection.php
    @@ -0,0 +1,121 @@
    +      'process' => [
    +        'target_id' => [
    +          [
    +            'plugin' => 'migration_lookup',
    +            'migration' => 'd7_field_collection',
    +            'source' => 'value',
    +          ],
    +          [
    +            'plugin' => 'extract',
    +            'index' => [0],
    +          ],
    +        ],
    +        'target_revision_id' => [
    +          [
    +            'plugin' => 'migration_lookup',
    +            'migration' => [
    +              'd7_field_collection_revisions',
    +              'd7_field_collection',
    +            ],
    +            'source_ids' => [
    +              'd7_field_collection_revisions' => ['revision_id'],
    +              'd7_field_collection' => ['value'],
    +            ],
    +            // D8.4 Does not like an empty source value, Even when using ids.
    +            'source' => 'value',
    +          ],
    +          [
    +            'plugin' => 'extract',
    +            'index' => [1],
    +          ],
    +        ],
    +      ],
    

    I think all of this needs to be pulled out of the field plugin and added to the migration yml itself. Meaning, add a process plugin that wraps all of this logic and passes in a list of migration names. Then we can abstract/hide all this complex 'target_id'/'target_revision_id'/'migration_lookup'/'extract' mess. And this process plugin should land in ERR, not in paragraphs.

  13. +++ b/src/Plugin/migrate/process/FieldCollectionFieldSettings.php
    @@ -0,0 +1,28 @@
    +class FieldCollectionFieldSettings extends ProcessPluginBase {
    

    This can extend static_map.

  14. +++ b/src/Plugin/migrate/process/ParagraphsStripPrefix.php
    @@ -0,0 +1,35 @@
    +      throw new MigrateException("StripPrefix plugin missing prefix to strip");
    

    Usually we use the plugin id instead of the class name.

heddn’s picture

mikelutz’s picture

@heddn, alright, I will reroll again piecemeal, and we can focus on getting each piece committed in smaller chunks in order. #2944844: Update testing database fixture to allow for testing more complex interactions was committed this morning, which is going to help a lot. I agree with this strategy, the next step is getting https://www.drupal.org/project/paragraphs/issues/2911242 RTBC and committed, along with #2911243: Create field collections field plugin which can be done concurrently. I will reroll those with dedicated unit tests against the new fixture such that they don't depend on each other. They won't provide any functionality on their own, but once they are committed, we can reopen

Ultimately I need this for a commissioned project pretty quickly, which is why I ran with everything. While I've now got a patch I can use for my project, I would very much like to see this work given back and eventually committed, although I will likely have less time to spend on it going forward.

That said, I have a few comments/questions about the above feedback items.

#4, 11, and 12 are all tied together and deal with what destination plugin(s) to use for the actual paragraph entity itself, and I spent quite a bit of time going back and forth on the issue. The entity destination only provides the entity id as an identifier, the entity_revision destination only provides the revision_id and with an entity_reference_revisions field, we need to be able to access both the entity_id and revision_id from a migration lookup. My problem was that the entity_reference_revision destination seems to be fundamentally broken in that it will not allow you to create a new revision of an existing entity, nor will it allow you to create stub entities. Prior to overriding the class, I submitted #2944836Migrate destination plugin attempts to update new entities, preventing creation of stub entities but truthfully, I can't figure out why that destination was written the way it was originally, and while I feel like my logic makes more sense and the original is buggy, with no existing test coverage around that class, I can't prove my solution won't affect someone else's existing migration. So, this is why I chose to extend it, either as a temporary measure until we can come to consensus around #2944836, or, if I can't convince ERR to change, it would still work, or we could extend off of EntityRevision directly and implement the rest of the ERR destination logic so we weren't depending on ERR. I did reference that ERR issue at the top of the destination plugin.

The fact of the matter is this is the destination for the paragraph entity itself, not the field which references it, and that isn't really ERR's job. ERR should be concerned with the field, not the entity which the field references. If ERR is going to be opinionated as to how that entity is saved in a migration, then it should support stubs and creating new revisions, I think.

As far as #12 goes, I'm up in the air. migration_lookup does such a good job of trying to find the reference, and creating a stub if it doesn't exist, I feel like any attempt to abstract that away in ERR is going to be far more messy, and unnecessary since the core plugins exist specifically to do that work, and it again hits the issue of ERR being opinionated about the entity it references rather than focusing on the field. That being said, I don't care where the logic goes as long as it works.

Finally, with issue #6 on the strip prefix plugin, I've actually come around to the thought that we should just get rid of it and just replace it with substr. When I first started working on this issue, one of the first patches I pulled to start working from had the 'prefix: field_' in the yml file, and the logic in the source to strip it, but I don't know any situation where field collection fields don't start with field_. I can't think of any situation where I need the logic of specifying or checking that. Everywhere it's used, I really just want to strip the first 6 characters, and the further I get in this the bigger of a PITA it is trying to pass around that value, which is essentially a constant. Can you think of any situation where someone might want to change that value in a migration alter or otherwise?

heddn’s picture

Regarding ERR, that was just an oversight and copy/paste from what core does when that piece of code was written. I was pretty instrumental in its original implementation, there is test coverage for a first time creation. And I thought for a re-save too. But I could be wrong.

I'm +1 on using substr to replace. We have some precedent on doing that in other places and it seems fairly sane.

And want to say a HUGE thanks for all your work on this. I'd love to see it keep moving too. I've got some time to help move things along myself. So, on to #2911242: Create field collections source migrate plugin and #2911243: Create field collections field plugin

mikelutz’s picture

Doh! You are right, there is a test class for that, it was buried and I missed it. Well, on the positive side, my change still passes the original test then, which helps my case. I just need to add a test for what I saw was broken, so that's good.

StevenPatz’s picture

Hey guys I too have a client who is doing a migration. D7 has Field Collections, D8 will have a paragraph to hold that info. So any help with testing etc you need I can help with.

I'm currently, after patching, trying to get the migration to work as described above.

mikelutz’s picture

@StevenPatz That's awesome, I'm very interested in hearing how far you get with the latest patch.

StevenPatz’s picture

@mikelutz Currently I am building out the PHP that maps FC columns to Paragraph field. Hoping to get that run locally tomorrow ( the FC we have is like 15 fields so it's a bit time consuming )

mikelutz’s picture

The latest patch here should do all that automatically, although it may not apply against the latest -dev because I had part of it committed yesterday.

StevenPatz’s picture

@mikelutz Oh that's interesting. Let me test.

noel.delacruz’s picture

Hi, would like to ask which version of the module should I apply the patch to? I'm using the latest dev build and doing some test since I have a drupal 7 with field collections. When I tried running drush ms it only shows upgrade_d7_field_collection_type. Is that supposed to be it or am I missing something?

heddn’s picture

You asked 9 hours ago. At 5 hours ago, another piece of the pie #2911242: Create field collections source migrate plugin landed in the dev branch. Next up is #2911243: Create field collections field plugin. All of everything that is still remaining will need re-rolls (potentially complicated re-rolls). But the happy thing is that we are 2 or 3 issues from having full support for migrating paragraphs or field collections from D7 to D8.

mikelutz’s picture

mikelutz’s picture

For those interested, here's a reroll against the latest -dev. Work is being carried out in individual issue queues now, this includes some extra work on the content migration that isn't quite ready for commiting yet, but might be worth testing.

noel.delacruz’s picture

I'm getting an input should be an array error when I tried migrating my nodes to Drupal 8. Has anyone encountered this issue using the latest patch. Below is my YML configuration

uuid: 62eeb230-1df9-47da-9218-803b99203c21
langcode: en
status: true
dependencies: {  }
id: upgrade_d7_node_press_release
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
migration_group: migrate_drupal_7
label: 'Nodes (Press Release)'
source:
  plugin: d7_node
  node_type: press_release
process:
  nid: tnid
  vid: vid
  langcode:
    plugin: default_value
    source: language
    default_value: und
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
  comment_node_press_release/0/status: comment
  body:
    plugin: get
    source: body
  field_multimedia:
    plugin: sub_process
    source: field_multimedia
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: upgrade_d7_field_collection
          source: value
        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration:
            - upgrade_d7_field_collection_revisions
            - upgrade_d7_field_collection
          source_ids:
            d7_field_collection_revisions:
              - revision_id
            d7_field_collection:
              - value
          source: value
        -
          plugin: extract
          index:
            - 1
  field_press_photo:
    plugin: sub_process
    source: field_press_photo
    process:
      target_id: fid
      alt: alt
      title: title
      width: width
      height: height
destination:
  plugin: 'entity:node'
  default_bundle: press_release
migration_dependencies:
  required:
    - upgrade_d7_user
    - upgrade_d7_node_type
  optional:
    - upgrade_d7_field_instance
    - upgrade_d7_comment_field_instance
mikelutz’s picture

That actually looks like a migrate_upgrade bug. The code they run to add the upgrade_ prefix to the migration_lookups doesn't seem to set the keys for the source_ids., so your migration_lookup fails, and sends a scaler to the extract process, which expects an array. Adding the upgrade_ prefix for the keys there would help you, but we will need to file a bug with migrate_upgrate to get that fixed.

mikelutz’s picture

StevenPatz’s picture

I'll likely be trying out the latest -dev with patches this afternoon. Should I apply the latest patch here or individual issue patches ( those still in progress )

mikelutz’s picture

Give #40 here a shot. it's got 2911243: Create field collections field plugin and an initial version of #2911244: Field collections derriver and base migration rolled in. It's the most up to date/farthest along code that I've got.

StevenPatz’s picture

@mikelutz

Do I need to create Paragraph/add fields to it before running the migration? I have two yml files. One is called d7_field_collection_authors it maps source fields to destination fields. Doing a drush ms shows this after running it:

d7_field_collection_authors Idle 1701 1701 0 2018-02-20 16:07:31

I then modified my already built content type migration to do a migration_lookup of d7_field_collection_authors, this is what I see for drush ms:

d7_node_not_real_name Idle 304 297 0 2018-02-20 16:11:09

So it looks like the migrations are running okay. But when I spot check some of the not_real_name contents, I just see
"No paragraph added yet" and a button to add one.

I'm definitely on the -dev version and the patch from #40 applied cleanly.

StevenPatz’s picture

Just checked the messages table for the not_real_name and the 7 or so that failed have this:

97474878437b0aeb46dad72e68bc2ffb45e806994b73c8275d967eb502398453  1       Array index missing, extraction failed.
 25394635474b4b896e6656e54678709f20b2fd8389ca7cf5bad8c068e847fd3a  1       Array index missing, extraction failed.
 2e8223c44f36c72537400d11bdcdd92288969473237c9d2e94992746d1fbda71  1       Array index missing, extraction failed.
 3f8fe5e0785210212d7175fec3c56a77f5325b272df5ebd87460fca3c9eb47b3  1       Array index missing, extraction failed.
 689c5c36e6d34c97052c562b8f47327749f7717d3f919f9bbe74b8a53c1581a2  1       Array index missing, extraction failed.
 7b3a96729c94603f8f33d9261d8fcf27c29d9907801f8bae4431df325a3dc0ea  1       Array index missing, extraction failed.
 df05872c73ed8e0613ea158959df94d271b205eee37229a16d4104524b1b30f6  1       Array index missing, extraction failed.
mikelutz’s picture

@StephenPatz . The code was designed to run with the standard Drupal Migrate UI, so dependencies are a bit tricky if you are running custom migrations. It's tough without seeing the actual migrations you are working with, but the way I've been testing is to run migrations in the following order.

1) d7_field_collection_type from the module. This should create the actual paragraph type that cooresponds to the field collection
2) d7_field followed by d7_field_instance should load both the old field_collection field as a new entity_reference_revisions field targeting paragraphs, as well as load the field and field instances of the old field collection against the new paragraph created from 1. If you need to run a subset of these to just get the field collection fields, there are a bunch of processes from the paragraphs.module hook_migration_alter that you will need to add to make sure the field collection bundles translate over.
3) d7_view_modes, d7_field_instance_widget_settings, d7_field_formatter_settings. These are optional, but if you don't run them you will have to manually add those fields to the form mode and view mode.
4) d7_field_collection and d7_field_collection_revision to load all the new paragraphs. Those are derived into separate migrations for each field_collection, again, you can manually create ones for specific field_collections, just follow the format of the included ones and add the field_name to the source plugin.
5) Finally run d7_node, and if you are manually mapping the fields, the lookup process for the paragraph is a little complicated now, see Drupal/paragraphs/Plugin/migrate/field/FieldCollection::processFieldValues() to see how to set that up.

It does process from a migrate_lookup to an extraction, and your error message is something using the Extract process plugin without specifying the index, but all the migrations included in the patch definitely have a hard coded index array for each extract process, so I don't think that is coming from the patch.

noel.delacruz’s picture

Hello @StevenPatz, can you share with me your migration yml for the field collection and nodes you are trying to migrate. Currently doing multiple tests migrating/converting field collections to paragraphs

noel.delacruz’s picture

Hello @mikelutz, sorry which part in YML configuration should I add the upgrade_ prefix? Not familiar with the keys you are currently pointing.

I'm also currently trying your process. In Step #2 I'm getting this error when I try migrating my upgrade_d7_field_instance. The same error occurred when I try migrating upgrade_d7_field_instance_widget_settings and upgrade_d7_field_formatter_settings

Missing bundle entity, entity type paragraphs_type, entity id edia.                                                            [error]
(/Applications/XAMPP/xamppfiles/htdocs/foundation-eight/core/lib/Drupal/Core/Entity/EntityType.php:883)
Missing bundle entity, entity type paragraphs_type, entity id edia.                                                            [error]
(/Applications/XAMPP/xamppfiles/htdocs/foundation-eight/core/lib/Drupal/Core/Entity/EntityType.php:883)

Here is my migration yml for the field instance.

uuid: 492e538a-b77d-4208-907f-52a5905b561f
langcode: en
status: true
dependencies: {  }
id: upgrade_d7_field_instance
class: Drupal\migrate_drupal\Plugin\migrate\FieldMigration
field_plugin_method: processFieldInstance
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
migration_group: migrate_drupal_7
label: 'Field instance configuration'
source:
  plugin: d7_field_instance
  constants:
    status: true
process:
  type:
    plugin: process_field
    source: type
    method: getFieldType
  entity_type:
    -
      plugin: get
      source: entity_type
    -
      plugin: static_map
      map:
        field_collection_item: paragraph
        paragraphs_item: paragraph
      bypass: true
  field_name: field_name
  bundle:
    -
      plugin: static_map
      source: bundle
      bypass: true
      map:
        comment_node_forum: comment_forum
    -
      plugin: paragraphs_process_on_value
      source_value: entity_type
      expected_value: field_collection_item
      process:
        plugin: substr
        start: 6
  label: label
  description: description
  required: required
  status: constants/status
  settings:
    plugin: d7_field_instance_settings
    source:
      - settings
      - widget
      - field_definition
  default_value_function: ''
  default_value:
    plugin: d7_field_instance_defaults
    source:
      - default_value
      - widget
  translatable: translatable
destination:
  plugin: 'entity:field_config'
migration_dependencies:
  required:
    - upgrade_d7_field
  optional:
    - upgrade_d7_node_type
    - upgrade_d7_comment_type
    - upgrade_d7_taxonomy_vocabulary
    - upgrade_d7_field_collection_type
    - upgrade_d7_paragraphs_type
mikelutz’s picture

That error suggests the d7_field_collection_type migration had a problem. Did you have a field collection named field_edia? Did you have any skips or problems with that migration?

noel.delacruz’s picture

@mikelutz, The field collection field_edia doesn't exist and I didn't have any problems when I migrated upgrade_d7_field_collection_type. I actually have only one field collection on my drupal 7. I'm actually testing this on a vanilla instance of Drupal 7 and 8

Field Collection

https://imgur.com/hrB5KVg

Fields

https://imgur.com/Hm3D36R

mikelutz’s picture

Hmm, somehow it's stripping the first 6 characters off the bundle name twice. It should be changing field_multimedia to multimedia, but it's changing it to edia. I don't see why fof the top of my head, I'll have to look more closely in the morning. There should be tests against that.

noel.delacruz’s picture

Aside from that most of the fields successfully migrated. I guess it might take a while before we get patch that will allow us to migrate the content within th field collections.

StevenPatz’s picture

@MikeLutz I'll post the migrations in a bit, have to clean out any identifying remarks etc. Also will follow the updated steps you posted, I know that I've run d7_field_collection_type and that two FCs come over ( only need one ) in the description of the paragraph type I see "Migrated from field_collection field_notice_author" I think I'm close. I'll have an update in a few.

StevenPatz’s picture

This is migrate_plus.migration.d7_field_collection_authors.yml

langcode: en
status: true
dependencies: {  }
id: d7_field_collection_authors
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
migration_group: mbw_d7
label: Field Collection Authors
source:
  plugin: d7_field_collection_item
  key: migrate
#  field_name is used in our custom plugin to get data about this field_collection_item.
  field_name: field_notice_author
process:
  field_email:
    plugin: iterator
    source: field_email
    process:
      value: email
    revision_id: revision_id
  field_phone:
    plugin: iterator
    source: field_phone
    process:
      value: value
    revision_id: revision_id
  field_fax:
    plugin: iterator
    source: field_fax
    process:
      value: value
    revision_id: revision_id
  field_first_name:
    plugin: iterator
    source: field_abbr_name
    process:
      value: value
    revision_id: revision_id
  field_last_name:
    plugin: iterator
    source: field_short_name
    process:
      value: value
    revision_id: revision_id
  field_office:
    plugin: iterator
    source: field_official_name
    process:
      value: value
    revision_id: revision_id
  field_room:
    plugin: iterator
    source: field_event_room
    process:
      value: value
    revision_id: revision_id
  field_location_ref:
    plugin: iterator
    source: field_location_ref
    process:
      value: value
    revision_id: revision_id
destination:
  plugin: 'entity_reference_revisions:paragraph'
  default_bundle: notice_author
migration_dependencies:
  required: {  }
  optional: {  }
StevenPatz’s picture

This is the Content Type I am then trying to get the Migrated FC into:

field_notice_author:
    plugin: sub_process
    source: field_notice_author
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: d7_field_collection_authors
          source: value
        -
          plugin: extract
          index:
            - 0
mikelutz’s picture

@noel.delacruz . The latest patch is supposed to moving content over if the migrations are done correctly. I'm still not sure why yours is stripping 12 characters from the bundle instead of 6, I can't seem to duplicate the behavior in the tests.

mikelutz’s picture

@stevenPatz The field_notice_author process in your content type needs to return a target revision id in addition to a target id. (necessary for an entity_reference_revisions field, though I think that is actually a bug on the ERR side). If you aren't running a migration of revisions, and assuming only one copy of each paragraph, you could try:

field_notice_author:
    plugin: sub_process
    source: field_notice_author
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: d7_field_collection_authors
          source: value
        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration: d7_field_collection_authors
          source: value
        -
          plugin: extract
          index:
            - 1
StevenPatz’s picture

@MikeLutz Busy day today with this and other things BUT. Steps 1-3 of #48 worked for me. I have a new Paragraph type with the fields and widget items imported in. Steps 4-5 soon, but I'm feeling good about this. Great work on the changes.

noel.delacruz’s picture

@mikelutz, just to make sure I'm in the same page as everyone. I'm using the dev build of paragraphs module. I tried applying the latest patch and I think it didn't patch cleanly

curl https://www.drupal.org/files/issues/2897021-40.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch | patch -p1

patching file migrations/d7_field_collection.yml
patching file migrations/d7_field_collection_revisions.yml
patching file migrations/d7_paragraphs.yml
patching file migrations/d7_paragraphs_revisions.yml
patching file paragraphs.module
Hunk #1 FAILED at 5.
Hunk #2 succeeded at 445 (offset -1 lines).
1 out of 2 hunks FAILED -- saving rejects to file paragraphs.module.rej
patching file src/Plugin/migrate/D7FieldCollectionItemDeriver.php
patching file src/Plugin/migrate/D7ParagraphsItemDeriver.php
patching file src/Plugin/migrate/destination/Paragraphs.php
patching file src/Plugin/migrate/field/FieldCollection.php
patching file src/Plugin/migrate/field/Paragraphs.php
patching file src/Plugin/migrate/process/FieldCollectionFieldInstanceSettings.php
patching file src/Plugin/migrate/process/FieldCollectionFieldSettings.php
patching file src/Plugin/migrate/process/ParagraphsFieldInstanceSettings.php
patching file src/Plugin/migrate/process/ParagraphsFieldSettings.php
patching file src/Plugin/migrate/process/ParagraphsProcessOnValue.php
patching file src/Plugin/migrate/process/ProcessPluginBase.php
patching file tests/src/Kernel/ParagraphsMigrationTest.php
patching file tests/src/Unit/migrate/FieldCollectionFieldSettingsTest.php
patching file tests/src/Unit/migrate/FieldCollectionsFieldInstanceSettingsTest.php
patching file tests/src/Unit/migrate/ParagraphsFieldInstanceSettingsTest.php
patching file tests/src/Unit/migrate/ParagraphsFieldSettingsTest.php
patching file tests/src/Unit/migrate/ParagraphsProcessOnValueTest.php
patching file tests/src/Unit/migrate/ProcessTestCase.php
***************
*** 5,14 ****
   * Contains paragraphs.module
   */
  
- use Drupal\Core\Field\FieldConfigInterface;
  use Drupal\Core\Url;
  use Drupal\Core\Routing\RouteMatchInterface;
  use Drupal\field\FieldStorageConfigInterface;
  use Drupal\paragraphs\Entity\ParagraphsType;
  use Drupal\Core\Render\Element;
  
--- 5,19 ----
   * Contains paragraphs.module
   */
  
  use Drupal\Core\Url;
  use Drupal\Core\Routing\RouteMatchInterface;
  use Drupal\field\FieldStorageConfigInterface;
+ use Drupal\field\Plugin\migrate\source\d7\Field;
+ use Drupal\field\Plugin\migrate\source\d7\FieldInstance;
+ use Drupal\field\Plugin\migrate\source\d7\FieldInstancePerViewMode;
+ use Drupal\field\Plugin\migrate\source\d7\FieldInstancePerFormDisplay;
+ use Drupal\field\Plugin\migrate\source\d7\ViewMode;
+ use Drupal\migrate_drupal\Plugin\migrate\FieldMigration;
  use Drupal\paragraphs\Entity\ParagraphsType;
  use Drupal\Core\Render\Element;
  
mikelutz’s picture

noel.delacruz’s picture

@mikelutz, thank you the patch applied cleanly. By the way you mentioned on how to fix my issue regarding input should be an array and that I should add an upgrade_ prefix in the keys in order to fix the issue

Is this the part you mentioned that I should add the prefix in the source_ids section? Apologies for my ignorance since I'm still new with migration

target_revision_id:
  -
    plugin: migration_lookup
    migration:
      - upgrade_d7_field_collection_revisions
      - upgrade_d7_field_collection
    source_ids:
      d7_field_collection_revisions:
        - revision_id
      d7_field_collection:
        - value
    source: value
  -
    plugin: extract
    index:
      - 1
noel.delacruz’s picture

Just did another test today trying to convert my field_collections to paragraphs was unsuccessful in migrating the content. I really feel I'm close to trying to get this to work since @mikelutz said the latest patch will be able to migrate the content also but unable to figure where the issue is coming from and how do I fix it. I'll share here what I have accomplished so far.

Drupal Version - 8.4.5

Using Drush I download the necessary modules -

Migrate Plus (8.x-4.0-beta2)
Migrate Tools (8.x-4.0-beta2)
Drupal Upgrade / Migrate Upgrade (8.x-3.0-rc3)
Paragraphs (8.x-1.2+47-dev) with the latest patch #62

And then followed @mikeLutz steps in migrating the items in the following order

1.) Migrate upgrade_d7_field_collection_type. Didn't have any problems here
2.) Migrating the upgrade_d7_field was also successful but when I tried migrating the field instance (upgrade_d7_field_instance) I came across an error which @mikelutz cannot replicate. I'm using the latest patch here by the way. The error below also show itself when I tried migrating upgrade_d7_field_instance_widget_settings and upgrade_d7_field_instance_widget_settings. I only have one field collection registered on my Drupal 7

Field Collection - https://imgur.com/rtx4yaY

Missing bundle entity, entity type paragraphs_type, entity id s.                                                                                                [error]
(/Applications/XAMPP/xamppfiles/htdocs/foundation-eight/core/lib/Drupal/Core/Entity/EntityType.php:883)
Missing bundle entity, entity type paragraphs_type, entity id s.                                                                                                [error]
(/Applications/XAMPP/xamppfiles/htdocs/foundation-eight/core/lib/Drupal/Core/Entity/EntityType.php:883)

3.) Migrating upgrade_d7_view_modes was also clean, upgrade_d7_field_instance_widget_settings and upgrade_d7_field_instance_widget_settings had issue which is the same as above

4.) Before migrating my field collections I did a comparison with @StevenPatz YML configuration. I noticed some difference. Here is my YML configuration. On his YML configuration, the field plugin are set to iterator but mine is set to get, and the destination plugin is set to entity_reference_revisions:paragraph while my configuration is set to paragraph. Decided to follow his configuration hoping it might work. I also did the same for my upgrade_d7_field_collection_revisions_authors

migration_tags:
  - 'Drupal 7'
migration_group: migrate_drupal_7
label: 'Field Collections (Authors)'
source:
  plugin: d7_field_collection_item
  field_name: field_authors
process:
  type: bundle
  field_authors_name:
    plugin: get
    source: field_authors_name
  field_email:
    plugin: get
    source: field_email
destination:
  plugin: paragraphs
  default_bundle: authors
migration_dependencies:
  required:
    - upgrade_d7_field_collection_type
  optional:
    - upgrade_d7_field_instance

5.) Before migrating my content, I also did a comparison with @StevenPatz.

Below is my YML configuration. I noticed that my machine name is incomplete. It only shows upgrade_d7_field_collection, but shouldn't it be upgrade_d7_field_collection_authors which should be correct? Because that is the machine name of my field collection upgrade_d7_field_collection_authors.

body:
  plugin: get
  source: body
field_authors:
  plugin: sub_process
  source: field_authors
  process:
    target_id:
      -
        plugin: migration_lookup
        migration: upgrade_d7_field_collection
        source: value
      -
        plugin: extract
        index:
          - 0
    target_revision_id:
      -
        plugin: migration_lookup
        migration:
          - upgrade_d7_field_collection_revisions
          - upgrade_d7_field_collection
        source_ids:
          d7_field_collection_revisions:
            - revision_id
          d7_field_collection:
            - value
        source: value
      -
        plugin: extract
        index:
          - 1
 

6.) After making the modifications and changes and migrated the content type/node, the fields are empty as shown in this screenshot link below.

https://imgur.com/pNkYFTc

SQL Table:
https://imgur.com/JindD4r

Already tried creating a fresh drupal instance of 7 and 8. Already also tried re-installing the modules and applying the patch. But I still get the same result

mikelutz’s picture

@noel.delacruz

You are getting close. Still not sure on your error migrating instances, but if in d8, you have author and email field attached to your new paragraph type, then it is working. There may be an incompatibility with some other migration I'll need to look into, but I have a few revisions left to make on the field plugins so hopefully I'll find it with further testing. I did not do any testing with migrate_upgrade, I've been using drupal console's migrate:setup, and the migrate test classes for testing. migrate_upgrade does a bunch of altering of the migrations, which might be causing problems, such as the source_ids issue (yes, that is where you need to change the keys btw). I do need this to work with migrate_upgrade, so I want to identify any migrate_upgrade issues, but I'll need to work out whether they are bugs in migrate_upgrade or this code. I'll give it a shot with migrate_upgrade and see what happens.

You *shouldn't* need to change your machine name to add the _authors, since that is a derivative of the d7_field_collection migration, it should be pulled in automatically.

The 'paragraphs' destination plugin is one I wrote to work around a couple bugs in the entity_reference_revision:paragraphs destination plugin related to revisions and nested field collections. if you aren't worried about revisions and don't have nested field_collections, then ERR:paragraphs should be okay, but if you are doing a separate revisions migration, use the paragraphs destination, it will handle things correctly until ERR's bug is resolved.

Either get or iterator (now sub_process) should work for fields. If yours was generated by migrate_upgrade with get, then that should be fine.

So, at first glance, yours looks like it should have worked.

What does your paragraphs_item table look like? Did it create the entities but not fill them, or did it not create them at all?

StevenPatz’s picture

Updates, and some Progress.

So I decided to just run the built-in migrations so I ran d7_field_collection_revisions:notice_author etc instead of the custom yml, thinking if I can at least get the data into the D8 system, cleaning up would be easier. I then changed the custom Content Type migration to reference the built-in migrations:

notice_author:
    plugin: sub_process
    source: field_notice_author
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: d7_field_collection:notice_author
          source: value
        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration: d7_field_collection:notice_author
          source: value
        -
          plugin: extract
          index:
            - 1

This is drush ms:

d7_field_collection:language                 Idle    209    209       0            2018-02-22 11:26:34
d7_field_collection:notice_author            Idle    1701   1636      0            2018-02-22 11:21:58
d7_field_collection_revisions:notice_author  Idle    909    909       0            2018-02-22 11:28:10
d7_node_xxxx_notice                          Idle    304    292       0            2018-02-22 12:09:46

All looks okay ( the notice_author didn't fully do because the FAX number from D7 is plain text with letters and words in and in D8 I'm using telephone, but enough did migrate for testing )

I created a View to see what data is in the Paragraphs and I get a few pages of actual migrated content from D7. But when I spot check xxxx_notice nodes, I can't find any that have a paragraph attached like they should. I feel like I'm really close, but can't tell if it's still some code that needs added/fixed in the module or if it's just not the right yml settings for the migration. For the 12 that didn't migrate drush mmsg shows:

source_ids_hash                                                   level   message
 23ad491edbf8adc287ce9b8add0e54f7c347f607c143283d241528ecade43b65  1       Array index missing, extraction failed.
 49ff1fef7821a0f41974937d0e37e7b983249336ecdcbe39db46d914a0b590f2  1       Array index missing, extraction failed.
 79e1dcd27f70ab335171e6ee4c0d7a0019147403610544c5a169710af5e7ae23  1       Array index missing, extraction failed.
 7ed90ee65e0496b356c0fbca1f607a61dfd18863a2064f38ef889de05c8d504c  1       Array index missing, extraction failed.
 3f0b8eb5f902e36f7632e9c3a823dbce5811c566c66ce35a8ebb0af8277b87ca  1       Array index missing, extraction failed.
 50836b3270ef3c0a23bff651281383f1bf257a6958c886db3a622cb4d58c739f  1       Array index missing, extraction failed.
 746203f8f84a2e8604839ddc73fa7aff0fdbff7403c54fd1fa35e6554293cd42  1       Array index missing, extraction failed.
 abff778795884257152fdc75a1897e8b13b38fb4f8f06dc73cc2c679d3eb1c1f  1       Array index missing, extraction failed.
 d53a9484c848e53322e15583a7bb8619a1634b1f4387252c5c7bc02daa39bd9c  1       Array index missing, extraction failed.
 709c4b1c23f757b1768ab02f1187a8724c435262ae25f1686cb988d633ec079a  1       Array index missing, extraction failed.
 f96997f87855b6b105d1df6499c085ba255d689a024653b0a079cd6f92ce9d64  1       Array index missing, extraction failed.
 7c5c79d278b0f3e41cfc7304ba742a7d7b52ae388e34a3174d129dc226584842  1       Array index missing, extraction failed.

Should I apply the latest patch? Any other ideas/tips?

mikelutz’s picture

@StephenPatz You are getting really close. I think your extract error messages are due to #2890690: MigrateLookup plugin has inconsistent return values
Basically the paragraphs that are failing are causing a stub creation, which is returning only an entity_id to extract. So these errors are for the content items referencing the paragraph items that are missing anyway.

Honestly everything looks right, except in your content type process you mapped to 'notice_author' instead of 'field_notice_author'. That looks to have changed from what you posted previously, so I'm not sure if that made it into your migration, or was just an error in copying it here.

StevenPatz’s picture

[EDIT} Hmmm, though the machine name in the content type is field_notice_author, the Paragraph machine name is notice_author and the source machine name is field_notice_author. Let me re-run these again]

Yeah I doubled checked machine names this morning and notice_author is the name of the Paragraph, that was a previous typo and had caused a lot more of the nodes to not be migrated. Definitely need to get this working for our migration, so I'll update as I keep playing around.

rozh’s picture

mikelutz, You're amazing! Great job here.

I tried latest dev of Paragraphs with patch from #62. I used regular migration with core Migrate UI module but haven't succed.
Paragraph types and fields was created but content wasn't. Actually nodes with at least one field collection item wasn't migrated.

mikelutz’s picture

@StephenPatz yeah, it's a little complicated. the field name is 'field_notice_author', which is an entity reference revisions field, which references paragraph entities, and allows paragraphs of bundle type 'notice_author'. For field collections, the field name and bundle type were always the same string with a one to one relationship, but they are completely separate concepts in paragraphs.

@rozh I'm still working on the content migrations, I've gotten some to come over, but there are still issues being reported, so there is still work to do. Keep an eye here for updates. We are really pushing to have https://www.drupal.org/project/paragraphs/issues/2911243 reviewed and committed before I spend too much more time working on the content migrations.

StevenPatz’s picture

@MikeLutz in re #70 is there a separate issue I can look at/review/help with regarding content migrations? it seems like you're saying what I've experienced, Paragraph type gets created, fields in that type get created, and the data from FC is loaded, but....getting that Paragraph attached to a node is not working?

noel.delacruz’s picture

@mikelutz

Is there any difference with using drupal console's migrate:setup? I might want to try it just to get this working. Do you have any documentation on how I can use Drupal console's migrate. Do I also need any other migration module or dependencies?

mikelutz’s picture

@StevenPatz, I really need#2911243 reviewed and committed before I can put serious work into the content migrations. That is the issue that migrates the fields and field instances, and any content migration work depends on it. The patch here contains an earlier version of that patch along with some preliminary content work I did.

@noel.delacruz drupal consoles migrate:setup just sets the migrations up differently, doesn't try to change the migration ids and such. I'm not saying it's better or worse, just a different method. Good news on your side though, @heddn did post some steps to replicate in that thread I mentioned above that seem to trigger your field instance error, so I should be able to track it down soon.

colan’s picture

@mikelutz: This should help with referring to other issues, from the More information about text formats link, so that you don't have to add the links yourself:

References to project issues in the form of [#1234] (or [#1234-2] for comments) turn into links automatically, with the title of the issue appended. The status of the issue is shown on hover. If '@' is appended (e.g. [#1234@]), the user the issue is assigned to will also be printed.

#2911243: Create field collections field plugin, for example.

StevenPatz’s picture

@mikelutz I've applied the latest patch from https://www.drupal.org/project/paragraphs/issues/2911243 and see no errors ( still can't get the actual nodes with their paragraphs data). I'll be working on this over the weekend, I'll check back here to see if there's any progress.

noel.delacruz’s picture

@mikelutz

Drupal Console migrate doesn't generate the YML file for migration automatically right? Unlike Migrate Upgrade which can generate the configuration via drush durhs migrate-upgrade --configure-only and then export it using drush config-export --destination="path".

And If I'm going to drupal console migrate I need to also build my own module to migrate those item I have right?

-------------------------------------------------------------

Tried using drupal core migrate module, and I get the error array index missing, extraction failed when migrating my node

https://imgur.com/MHLsDjl

I believe the same issue happens if I try drush migrate-upgrade. These are nodes with field collections that are embedded.

drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven

The results differ if I tried using a different command on drush migrate-upgrade

If I tried following @mikelutz steps indicated above, I'm still getting input should be an array despite installing the latest release of migrate upgrade that addresses the issue on the source ids (Migration Lookup source_ids array keys are not properly prefixed when creating migrations.). Below is the YML file generated by migrate upgrade

source:
  plugin: d7_node
  node_type: press_release
process:
  nid: tnid
  vid: vid
  langcode:
    plugin: default_value
    source: language
    default_value: und
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
  comment_node_press_release/0/status: comment
  body:
    plugin: get
    source: body
  field_authors:
    plugin: sub_process
    source: field_authors
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: upgrade_d7_field_collection
          source: value
        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration:
            - upgrade_d7_field_collection_revisions
            - upgrade_d7_field_collection
          source_ids:
            upgrade_d7_field_collection_revisions:
              - revision_id
            upgrade_d7_field_collection:
              - value
          source: value
        -
          plugin: extract
          index:
            - 1
destination:
  plugin: 'entity:node'
  default_bundle: press_release
migration_dependencies:
  required:
    - upgrade_d7_user
    - upgrade_d7_node_type
  optional:
    - upgrade_d7_field_instance
    - upgrade_d7_comment_field_instance
noel.delacruz’s picture

Ok, let me compile/summarize the test I have done so far in trying to migrate/convert the field collection to paragraphs using the available resources/modules and patches

Drupal - 8.4.5
Drupal - 7.56

Modules

1.) Migrate Upgrade (8.x-3.x-dev)
2.) Migrate Tools (8.x-4.0-beta3)
3.) Migrate Plus (8.x-4.0-beta3)
4.) Paragraphs (8.x-1.x-dev) with latest patch applied

Test 1

I tried migrating the content and fields using the following steps below via drush

1.) drush migrate-upgrade --configure-only
2.) drush mim -y --all

I'm still getting the error Missing bundle entity when migrating field_instance, field_formatter_settings, field_instance_widget_settings.

I noticed that the node that contains field collection failed to migrate and rendered a migration message Input should be an array.. Below is the screenshot that contains the error within the SQL

https://imgur.com/f85q7V5

The paragraphs_item table were filled as shown in this screenshot.

https://imgur.com/48blUov

Data was not migrated successfully as shown in the screenshot. As you can the tables were empty.

Field Authors Name - https://imgur.com/YLeTJBR
Field Email - https://imgur.com/hyVT8kc

The fields within the field collection was migrated but did not attach to the paragraph types that was created. You have to attach those fields manaully, screenshot below

https://imgur.com/yO64RJu

----------------------------------------

Test 2

For the second test, I used a single command to migrate or upgrade all field and entities from Drupal 7 to Drupal 8 drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven and it had a different result

The node or content type that contains the field collection was not migrated successfully and had a different error compared to the first test. It showed an error "Array index missing, extraction failed". You can see in the screenshot below.

https://imgur.com/SVW3Q0f

Contents or Data within the fields from the field collection was reflected on the paragraphs field table. As you can it has the same data as the node in drupal 7

Author Email Field - https://imgur.com/Vw3DNko

Author Name Field - https://imgur.com/ktnq2Ed

Paragraphs Item Table - https://imgur.com/Th5VU7I

Paragraphs Item Field Data Table - https://imgur.com/XMLaLUt

Drupal 7 Node - https://imgur.com/XzWXD03

The Field have also been attached to the Paragraph Types

https://imgur.com/J2annAr

----------------------------------------

Test 3

For the third test, what I did was to configure or modify the YML configuration for the node in order Input is an array
and array index missing. Exported the configuration drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven

Edit the node YML configuration that has paragraphs/field collection config

 body:
    plugin: get
    source: body
  field_authors:
    plugin: sub_process
    source: field_authors
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: upgrade_d7_field_collection
          source: value
        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration:
            - upgrade_d7_field_collection_revisions
            - upgrade_d7_field_collection
          source_ids:
            upgrade_d7_field_collection_revisions:
              - revision_id
            upgrade_d7_field_collection:
              - value
          source: value
        -
          plugin: extract
          index:
            - 1
destination:
  plugin: 'entity:node'
  default_bundle: press_release
migration_dependencies:
  required:
    - upgrade_d7_user
    - upgrade_d7_node_type
  optional:
    - upgrade_d7_field_instance
    - upgrade_d7_comment_field_instance

Remove this part within the configuration

        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration:
            - upgrade_d7_field_collection_revisions
            - upgrade_d7_field_collection
          source_ids:
            upgrade_d7_field_collection_revisions:
              - revision_id
            upgrade_d7_field_collection:
              - value
          source: value
        -
          plugin: extract
          index:
            - 1

The result was it managed to migrate the node but the data from the field collection converted to paragraphs did not attach to the node.

Not sure what else I need to do in order to make this work since I have D7 Project that contains field collection. If you have any more ideas or suggestion that I please share it with me so I can do some tests. I tried using Drupal Console migrate:setup, but after the part of entering your local directory Local file directory containing your source site nothing happened did I miss anything?

Drupal Console migrate:setup - https://imgur.com/4BkbBxh

Drupal Console migrate:execute - https://imgur.com/7dq2Y6i

mikelutz’s picture

@noel.delacruz Wow!, thank you so much for the detailed data. I am back looking into this today to track down some of the issues you and StephenPatz have been having. Looking at the error messages, I suspect we are running into issues with creating paragraph stubs and properly updating them, and I still think migrate_upgrade is doing something as well, but between your report and one provided by @heddn in #2911243: Create field collections field plugin I should have enough information to duplicate the issues in the debugger and see what's going on. Stay tuned.

mikelutz’s picture

Here's a new patch for you guys to chew on. It should solve the issue with field instance errors, though I'm not really happy with how it does it at the moment. You would need to start fresh and rerun migrate-upgrade for it to work.

StevenPatz’s picture

@mikelutz Hoping to apply patch later today or first thing in the morning.

mikelutz’s picture

StevenPatz’s picture

@mikelutz That last patch did it. I have my Content Type with a Paragraph attached with the actual data migrated from FC to Paragraph. My end is not nearly done as I'll need to make a custom migration ( was using all built-ins to help here and to prove that it could be done) But your work ( as well as others) is coming along quite well.

noel.delacruz’s picture

I'm getting Input should be an array when I tried migrating my node that contains field_collection config. I'm using the latest release of migrate upgrade

uuid: 5d915473-87fe-43b3-9ba3-d1837315bfcd
langcode: en
status: true
dependencies: {  }
id: upgrade_d7_node_press_release
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
  - paragraphs-processed
migration_group: migrate_drupal_7
label: 'Nodes (Press Release)'
source:
  plugin: d7_node
  node_type: press_release
process:
  nid: tnid
  vid: vid
  langcode:
    plugin: default_value
    source: language
    default_value: und
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
  comment_node_press_release/0/status: comment
  body:
    plugin: get
    source: body
  field_authors:
    plugin: sub_process
    source: field_authors
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: upgrade_d7_field_collection
          source: value
        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration:
            - upgrade_d7_field_collection_revisions
            - upgrade_d7_field_collection
          source_ids:
            upgrade_d7_field_collection_revisions:
              - revision_id
            upgrade_d7_field_collection:
              - value
          source: value
        -
          plugin: extract
          index:
            - 1
destination:
  plugin: 'entity:node'
  default_bundle: press_release
migration_dependencies:
  required:
    - upgrade_d7_user
    - upgrade_d7_node_type
  optional:
    - upgrade_d7_field_instance
    - upgrade_d7_comment_field_instance
mikelutz’s picture

@noel.delacruz Are you sure the actual field collection migration is running first? There is an issue with the stub creation returning non-arrays to extract. I'll have it worked out in the final migrations, but for now, make sure the field collection migration runs first.

noel.delacruz’s picture

@mikelutz

Yes, these are the steps that I have taken

1.) Drush migrate-upgrade --configure-only
2.) Import Migrate Configurations drush cex
3.) Drush Migrate Import the items in the following order:
-- upgrade_d7_field_collection_type
-- upgrade_d7_user_role
-- upgrade_d7_user
-- upgrade_d7_field
-- upgrade_d7_field_instance
-- upgrade_d7_view_modes
-- upgrade_d7_field_formatter_settings
-- upgrade_d7_field_instance_widget_settings
-- upgrade_d7_field_collection_authors
-- upgrade_d7_field_collection_revisions_authors
-- upgrade_d7_node_press_release

If I try migrating using a single command from migrate-upgrade this is the error that I receive Array index missing, extraction failed.

drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven

By the way how long will it take in order to sort the array issue? I know there is a time difference so I'll try to get up as early as I can for testing. Really need this, so far field_instance have been migrated cleanly. Thank you for that..

mikelutz’s picture

StevenPatz’s picture

@mikelutz

Thanks for keeping this issues patches up to date. This latest applied cleanly for me, and I am in the process of making a custom migration with all of this work ( because we don't need a lot of the fields the built in migrations do.)

I'll update as I move forward.

noel.delacruz’s picture

Will try the latest patch and use and clean drupal instance. Going to use migrate-upgrade since it is the only tool I know how to use for migration

noel.delacruz’s picture

Applied the new patch and I'm getting some new notifications.

curl https://www.drupal.org/files/issues/2897021-86.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch | patch -p1

------------------

Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
2 out of 2 hunks ignored -- saving rejects to file paragraphs.module.rej
patching file src/Plugin/migrate/D7FieldCollectionItemDeriver.php
patching file src/Plugin/migrate/D7ParagraphsItemDeriver.php
patching file src/Plugin/migrate/destination/Paragraphs.php
The next patch would create the file src/Plugin/migrate/field/FieldCollection.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/field/FieldCollection.php.rej
The next patch would create the file src/Plugin/migrate/field/Paragraphs.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/field/Paragraphs.php.rej
The next patch would create the file src/Plugin/migrate/process/FieldCollectionFieldInstanceSettings.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/process/FieldCollectionFieldInstanceSettings.php.rej
The next patch would create the file src/Plugin/migrate/process/FieldCollectionFieldSettings.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/process/FieldCollectionFieldSettings.php.rej
The next patch would create the file src/Plugin/migrate/process/ParagraphsFieldInstanceSettings.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/process/ParagraphsFieldInstanceSettings.php.rej
The next patch would create the file src/Plugin/migrate/process/ParagraphsFieldSettings.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/process/ParagraphsFieldSettings.php.rej
The next patch would create the file src/Plugin/migrate/process/ParagraphsProcessOnValue.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/process/ParagraphsProcessOnValue.php.rej
The next patch would create the file src/Plugin/migrate/process/ProcessPluginBase.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/Plugin/migrate/process/ProcessPluginBase.php.rej
patching file src/Plugin/migrate/source/d7/FieldCollectionItem.php
Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
3 out of 3 hunks ignored -- saving rejects to file src/Plugin/migrate/source/d7/FieldCollectionItem.php.rej
patching file src/Plugin/migrate/source/d7/FieldCollectionType.php
Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
3 out of 3 hunks ignored -- saving rejects to file src/Plugin/migrate/source/d7/FieldCollectionType.php.rej
The next patch would delete the file tests/src/Kernel/ParagraphsMigrationTest.php,
which does not exist!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored
patching file tests/src/Kernel/migrate/ParagraphContentMigrationTest.php
The next patch would create the file tests/src/Kernel/migrate/ParagraphsFieldMigrationTest.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Kernel/migrate/ParagraphsFieldMigrationTest.php.rej
The next patch would create the file tests/src/Kernel/migrate/ParagraphsMigrationTestBase.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Kernel/migrate/ParagraphsMigrationTestBase.php.rej
The next patch would create the file tests/src/Kernel/migrate/ParagraphsTypeMigrationTest.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Kernel/migrate/ParagraphsTypeMigrationTest.php.rej
The next patch would create the file tests/src/Unit/migrate/FieldCollectionFieldSettingsTest.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Unit/migrate/FieldCollectionFieldSettingsTest.php.rej
The next patch would create the file tests/src/Unit/migrate/FieldCollectionsFieldInstanceSettingsTest.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Unit/migrate/FieldCollectionsFieldInstanceSettingsTest.php.rej
The next patch would create the file tests/src/Unit/migrate/ParagraphsFieldInstanceSettingsTest.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Unit/migrate/ParagraphsFieldInstanceSettingsTest.php.rej
The next patch would create the file tests/src/Unit/migrate/ParagraphsFieldSettingsTest.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Unit/migrate/ParagraphsFieldSettingsTest.php.rej
The next patch would create the file tests/src/Unit/migrate/ParagraphsProcessOnValueTest.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Unit/migrate/ParagraphsProcessOnValueTest.php.rej
The next patch would create the file tests/src/Unit/migrate/ProcessTestCase.php,
which already exists!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file tests/src/Unit/migrate/ProcessTestCase.php.rej
mikelutz’s picture

#2911243: Create field collections field plugin was committed a couple hours ago which contains part of this patch, this patch will need a complete re-roll now. Fortunately that's the last blocker before I can work on the content import part, so I can get started on that again tomorrow.

noel.delacruz’s picture

Just a quick update, I followed @mikelutz steps mentioned in migration so far these are the results.

I'm using migrate-upgrade by the way. Fetch the migration the configuration first before starting. drush migrate-upgrade --configure-only

1.) Migrated first upgrade_d7_field_collection_type - Migration was clean
2.) Migrated some of the dependencies and requirements before proceeding to migrating the fields. These d7_user_role, d7_user, d7_comment_type, d7_node_type, d7_taxonomy_vocabulary
3.) Migrated d7_fields - Migration was clean
4.) Proceeded to migrate d7_field_instance, d7_view_modes, d7_field_instance_widget_settings, d7_field_formatter_settings. So far everything was clean
5.) Migrated the collection fields d7_field_collection_authors, d7_field_collection_revisions_authors. Migration was clean
6.) Lastly d7_node_press_release which is basically the node. There was no errors during the migration and I didn't get the Input should be an array error

Upon checking the data within the fields have been migrated on the SQL table as shown in the screenshot below

Authors Field - https://imgur.com/dDRo9Mx

Email Field - https://imgur.com/kZ5utbt

But upon checking the node. The fields were empty

https://imgur.com/3MjpPo9

https://imgur.com/3MjpPo9

Just a note, I created a new drupal 8 instance using 8.4.5. Downloaded the migration modules and applied the patch for the paragraphs-dev build. Also cleared the cache

mikelutz’s picture

Now that the blockers are in, I'm moving the remaining work to the appropriate child issue - #2911244: Field collections deriver and base migration. I'm going to re-roll what's left of this patch there and keep working.

mikelutz’s picture

noel.delacruz’s picture

Hi @mikelutz, do I need to apply first the patch here then apply the patch from #2911244: Field collections derriver and base migration?

mikelutz’s picture

Just use #2911244: Field collections deriver and base migration on the latest -dev now. Everything else has been committed.

miro_dietiker’s picture

There's a new 8.x-1.5 ERR release now so hopefully this should unlock progress in further Migration improvements.

davidwhthomas’s picture

Great to see the updates in this area.

I just want to note, I'm doing quite a bit of custom migration work and found I could use the following migration YML files to migrate the paragraph data, without the migration lookups. I deleted all the paragraphs data first with drupal console: drupal entity:delete paragraph --all and migrated with the same ID values as the d7 source DB.

This includes migration of the field collection item_id to the paragraph id and the reference to the paragraph on the host entity field.

Paragraph item migration

# YAML file for paragraph item: application_author
langcode: en
status: true
dependencies: {  }
id: upgrade_d7_field_collection_application_author
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
  - 'Field Collection Content'
label: 'Field Collections (Application author)'
source:
  # Drupal 7 database key in settings.php
  key: drupal_7_database
  plugin: d7_field_collection_item
  field_name: field_application_author
process:
  # Maps source field collection ID to paragraph ID
  id:
    plugin: get
    source: item_id
  revision_id:
    plugin: get
    source: revision_id
  field_application:
    plugin: sub_process
    source: field_application
    process:
      target_id: tid
  field_author_bio:
    plugin: get
    source: field_author_bio
  field_author_image:
    plugin: sub_process
    source: field_author_image
    process:
      target_id: fid
      alt: alt
      title: title
      width: width
      height: height
destination:
  plugin: 'entity_reference_revisions:paragraph'
  default_bundle: application_author

Host entity migration

# YAML for taxonomy term: Author
langcode: en
status: true
id: upgrade_d7_taxonomy_term_author
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
label: 'Taxonomy terms (Author)'
source:
  # Drupal 7 database key in settings.php
  key: drupal_7_database
  plugin: d7_taxonomy_term
  bundle: author
  track_changes: true
  constants:
    bundle: author
process:
  tid: tid
  vid: constants/bundle
  name: name
  description/value: description
  description/format: format
### Relevant paragraph field
  field_application_author:
    plugin: sub_process
    source: field_application_author
    process:
      target_id: value
      target_revision_id: revision_id
destination:
  plugin: 'entity:taxonomy_term'
  default_bundle: author

These migration configs can be imported via the config single import UI, or from the module migrations folder. They can then be run with drush:

drush migrate-import upgrade_d7_field_collection_application_author --feedback="10 items"
drush migrate-import upgrade_d7_taxonomy_term_author --feedback="10 items" --update

Maybe useful for others in a similar situation.

In the meantime, keep up the great work!

Jumoke’s picture

What's the correct format for migrating fc into a paragraph on a node?
This works but I am now noticing that the results are widely inconsistent, sometimes the data simply doesn't make it in. And in cases where I have multiple FC fields on a nodetype, it's assigning a FC into the wrong Paragraph! Scary!

  field_external_systems:
    plugin: sub_process
    source: field_external_systems
    process:
      target_id: value
      target_revision_id: revision_id

This format returns a "value" is not a valid entity message

  field_external_systems:
      plugin: iterator
      source: field_external_systems
      process:
        target_id:
          plugin: migration
          migration: usp_d7_field_collection_external_systems
          source: value
        target_revision_id: revision_id

Same as below:

#  field_external_systems:
    plugin: iterator
    source: field_external_systems
    process:
      target_id:
        plugin: migration
        migration: usp_d7_field_collection_external_systems
        source: '0'
      target_revision_id:
        plugin: migration
        migration: usp_d7_field_collection_external_systems
        source: '1'

My FC Migration into Paragraph Type looks like this:

id: usp_d7_field_collection_external_systems
label: D7 Field Collection - External Systems
migration_tags:
  - Drupal 7
  - 'Field Collection Content'
migration_group: usp_migrate
source:
  plugin: d7_field_collection_item
  field_name: field_external_systems
process:
  id:
    plugin: get
    source: item_id
  revision_id:
    plugin: get
    source: revision_id
  field_system_acronym:
    plugin: get
    source: field_system_acronym
  field_system_url:
    plugin: iterator
    source: field_system_url
    process:
      uri:
        -
          plugin: FixUrl
          source: url
      title: title
      options: attributes
destination:
  plugin: entity_reference_revisions:paragraph
  default_bundle: external_systems
migration_dependencies:
  required: {  }
  optional: {  }
dependencies:
  enforced:
    module:
      - migrate_drupal
      - migrate_tools
      - paragraphs
mikelutz’s picture

A side effect of the ERR destination, (and maybe one we should change) is that if it's a new entity, it enforces a new revision. So in your initial migration, you are keeping your ids, but your revision ids are probably changing, and need to be looked up. Check your database and see if the actual revision_ids match or not. I expect they won't. Try something like:

 field_external_systems:
    plugin: sub_process
    source: field_external_systems
    process:
      target_id: value
      target_revision_id:
	  	process:
		  -
		    plugin: migration
		    migration: usp_d7_field_collection_external_systems
            source: value
		  -
		    plugin: extract
		    index:
		      - 1
Jumoke’s picture

Thank you Mike for your help. I have some progress with the format below as suggested. Only thing now is that it brings in old revisions.
FC Migration

id: usp_d7_field_collection_external_systems
label: D7 Field Collection - External Systems
migration_tags:
  - Drupal 7
  - 'Field Collection Content'
migration_group: usp_migrate
source:
  plugin: d7_field_collection_item
  field_name: field_external_systems
process:
  id:
    plugin: get
    source: item_id
#  revision_id:
#    plugin: get
#    source: revision_id
  field_system_acronym:
    plugin: get
    source: field_system_acronym
  field_system_url:
    plugin: iterator
    source: field_system_url
    process:
      uri:
        -
          plugin: FixUrl
          source: url
      title: title
      options: attributes
destination:
  plugin: entity_reference_revisions:paragraph
  default_bundle: external_systems
migration_dependencies:
  required: {  }
  optional: {  }
dependencies:
  enforced:
    module:
      - migrate_drupal
      - migrate_tools
      - paragraphs

Node Migration

  field_external_systems:
    plugin: sub_process
    source: field_external_systems
    process:
      target_id: value
      target_revision_id:
          -
            plugin: migration_lookup
            migration: usp_d7_field_collection_external_systems
            source: value
          -
            plugin: extract
            index:
              - 1

You'd notice I have the revisions part commented out in the FC migration. With or without it, it's currently bringing in the wrong revisions. What else am I missing here? Looks like it's picking up random revisions.

mirsoft’s picture

I posted here a working solution to attach a parent entity to any existing paragraph or field collection that works for me (latest Drupal 8.6.2) with an assumption that I don't need to migrate the old paragraphs / field collection revisions, just attach the latest revision: https://www.drupal.org/project/entity_reference_revisions/issues/2990527...

Maybe it will help.

crash_springfield’s picture

What's the current status of this? Applying one patch I still get `The "field_collection_item" plugin does not exist.` while applying another patch I get `Plugin (d7_paragraphs) deriver Drupal\paragraphs\Plugin\migrate\D7ParagraphsItemDeriver does not exist`.

agudivad’s picture

Does anyone found working patch?

juampynr’s picture

I did a successful migration using #2911244: Field collections deriver and base migration.

sukh.singh’s picture

@juampynr it will be great if you can in detail mention how did you migrate field collections from D7 to D8?

cgalli’s picture

Great work! It is a bit confusion for the non involved, though....

Could anybody please post examples of the needed config files to get a field_collection (D7) to a paragraph field (D8). That would be very helpful...

Many thanks

juampynr’s picture

Sure @sukh.singh! I am on holiday now but next week I will add some notes here. Here is a hint though:

I installed the dev version of paragraphs, patched it, and then I used migrate upgrade and migrate drupal to generate migration entities, then I ran config migrations, which created paragraph types for each field collection, and finally I ran content migrations, which migrated content from field collections to paragraphs.

I posted an article with some of the commands involved at https://www.lullabot.com/articles/overview-migrating-drupal-sites-8.

bob.hinrichs’s picture

I ran into a migration error during field_formatter_settings import.. "The "field_collection_fields" plugin does not exist." This is in a display setting for the node type, when one selects "Fields Only" as the display for the field collection. I don't know if there is an applicable value that could be mapped for that formatter?

Here was my solution that prevents the migration from halting. In the configuration for d7_field_instance_per_view_mode, I added under the options/type map, under the values for field_collection:
field_collection_fields: entity_reference_entity_view

bob.hinrichs’s picture

I am using the recommended patch for Drupal 8.7 and I have hit a wall with it because of this little issue:

We are changing the machine name of the field in the node, that holds the D7 field collection that is becoming a paragraph on D8. This is done by changing the name via the yml configuration (uses string replace or static map plugin, etc.). In the case of the field name, this has created this problem:
1) The field machine name gets changed for the destination, when it is importing the field.
2) In FieldCollectionFieldInstanceSettings, the target bundle to be compared against is using row->getSourceProperty('field_name') as the target. However in our case the target's field name is changed, and the source no longer is accurate for this comparison. It should be comparing against the destination's not the source's values for the field name.

Going further, I encountered another, different problem.
I tried to fix this by overriding the class and setting my class's ID to use for field instance settings in my yml. This does not work though because the ids are hard-coded in FieldCollection class, in alterFieldMigration methods. I should be able to configure the plugins to use in the yml.

Sseto’s picture

@Juampy. I was wondering if you ever had a chance to write an article on migrating field collections from D7 to D8? Thanks!

juampynr’s picture

@Sseto hi! I followed this process, where I patched Field Collection module.

Sseto’s picture

Thanks @juampynr!

j.kop’s picture

@juampynr Hi! Is it also possible to migrate field collections inside D8? We currently use field collections 8.x-1.0-alpha1 and need to migrate a huge load of field collections to paragraphs.

I am not sure about how this differs from your nicely described D7 to D8 migration process as I do not know about the differences between D7 and D8. Could you give me a hint?

zorax’s picture

I try to migrate field_collection d7 to paragraph d8.
I use this patch.
When I migrate upgrade_d7_field_formatter_settings, I have error messages :
[error] The "field_collection_fields" plugin does not exist. like #108
In the configuration for d7_field_instance_per_view_mode, I added under the options/type map, under the values for field_collection:
field_collection_fields: entity_reference_entity_view.

field_collection:
          field_collection_view: entity_reference_revisions_entity_view
          field_collection_fields: entity_reference_entity_view

Now errors messages disappears, but field_collection are not imported....
Is there someting I miss ?
1 - patch paragraph with #121
2 - modify d7_field_formatter_settings
3 - Do I need someting else ?

iuliad’s picture

Hello,
Could I use this migration to migrate a single field from my field collection (which has only 1 entry per node, no multiple values), over to a single field on my node? Eg. my field collection has 1 field, a link, and now I have a single link field on my node and I would like to copy the data from the field collection directly into that field, without migrating the field collection into a paragraph. Thank you in advance.

DamienMcKenna’s picture

@iuliad: You could use the code in this patch to help you work out how but it won't do that sort of data structure change for you.

juampynr’s picture

@j.kop: for Field Collections to Paragraphs, see #2911244: Field collections deriver and base migration

iuliad’s picture

Hello, I have managed to do this in the following way:
Migrate data from a content-type with one field collection attaches that has single values (no multiple values) to normal, single-value fields on a content-type:
1. Create a custom source plugin that adds the field collection's field values to the node source

namespace Drupal\custom_module\Plugin\migrate\source\d7;

use Drupal\migrate\Row;
use Drupal\node\Plugin\migrate\source\d7\NodeComplete;

/**
 * Gets all node revisions from the source, including field collection data.
 *
 * @MigrateSource(
 *   id = "d7_node_complete_field_collection",
 *   source_module = "node"
 * )
 */
class NodeFieldCollection extends NodeComplete {

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {

    // Attach field collection values.
    $fieldCollectionName = $row->getSourceProperty('field_collection_name');
    $nid = $row->getSourceProperty('nid');
    // Get field collection.
    $fieldItems = $this->getFieldValues('node', $fieldCollectionName, $nid);

    // Get Field API field values.
    $field_names = array_keys($this->getFields('field_collection_item', $fieldCollectionName));
    if (isset($fieldItems) && isset($fieldItems[0]) &&
      is_array($fieldItems[0]) && isset($fieldItems[0]['value']) && isset($fieldItems[0]['revision_id'])) {
      $item_id = $fieldItems[0]['value'];
      $revision_id = $fieldItems[0]['revision_id'];

      foreach ($field_names as $field_name) {
        $value = $this->getFieldValues('field_collection_item', $field_name, $item_id, $revision_id);
        $row->setSourceProperty($field_name, $value);
      }
    }

    parent::prepareRow($row);
  }

}

2. Edit the content type's migration file to include the field collection field name and field values:

source:
  plugin: d7_node_complete_field_collection
  field_collection_name: field_resource
  node_type: article
process:
  field_link:
    -
      plugin: sub_process
      source: field_link
      process:
        uri: url
        title: title
        options: attributes

In this example, field_link is now a simple field on the new Drupal 8 node, and we will migrate the data from the Drupal 7 field collection's field field_link.

kevinquillen’s picture

Just ran an inline migration from D7 > D9. I can see that field collections were recreated as Paragraphs - however:

  • Form display settings were not set
  • Entity display settings were not set
  • No data was migrated

I read through the thread, but where can I start with getting data migrated in after I ran an inline upgrade?

jasonyarrington’s picture

Following. I am in the same situation as #119

iancawthorne’s picture

@kevinquillen and @jasonyarrington have a look at issue https://www.drupal.org/project/paragraphs/issues/3145755 patch #28 and also have a read of my comment at #34.

livomex533’s picture

I came from a child thread of this here https://www.drupal.org/project/paragraphs/issues/3020433
Are there any examples for a D8 Field Collection > D8 Paragraphs module migration?

lrocha’s picture

+1 on #122 comment

Wim Leers’s picture

RE all the comments wrt field collections: see #3020433-12: Migration from D8 Field Collection to D8 Paragraphs.