Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
Problem/Motivation
We need a method to upgrade/migrate from D7 field collections to D8 paragraphs.
Proposed resolution
Remaining tasks
- field collection type => paragraphs type migration, look at d7_node_type as a pattern
- a field collections source entity plugin, look at node source
- a field collections field plugin, look at entity reference
- field collections => paragraphs derriver and base migration, look at node as a pattern
User interface changes
API changes
Data model changes
Original
I've created a process plugin that allows for a mostly easy migration map for field_collection in Drupal 7 to paragraphs in Drupal 8. It was used successfully on a project recently, but it still needs more work.
I will be uploading a patch for the class itself, an example class, and some documentation shortly.
Comment | File | Size | Author |
---|
Comments
Comment #2
cruno CreditAttribution: cruno commentedHere's the patch.
Comment #3
cruno CreditAttribution: cruno commentedComment #4
DamienMcKennaComment #5
heddnThis isn't necessary. Assign the migration to a group that has the db key stored in it.
This should be a field plugin, not a process plugin. Look at date or link field plugins in 8.4.x
Comment #6
heddnThis is only part of the solution we also need a source deriver too.
Comment #7
heddnI worked out a plan with @Adita yesterday on what is needed here.
Comment #8
heddnChild issues opened.
Comment #9
elgordogrande CreditAttribution: elgordogrande commentedLooks like there's a namespace typo in the class: "paragraphss"
Comment #10
DamienMcKennaFixed the typoo.
Comment #11
colanYou got that one, but "usage" is still spelled wrong. ;)
Comment #12
docans CreditAttribution: docans commentedSo i applied the patch to my paragraph module manually because if you try to apply the patch using git apply -v path_name.patch , it does not work.
I run the
drush migrate-upgrade --legacy-db-url=mysql://user:password@server/db --legacy-root=http://example.com --configure-only
so as to generate the migration script but i didnt see anything related to field collection migration.Any idea what i did wrong here. Also what are the steps i am suppose to follow in order to successfuly migrate field collection in Drupal 7 to Paragraphs in Drupal 8 after applying the patch paragraphs-n2897021-10.patch
Thanks.
Comment #13
docans CreditAttribution: docans commentedComment #14
colanThis is definitely not critical as it's possible to use Paragraphs if you're not migrating D7 Field Collections. Critical means that the module doesn't work at all, under any conditions.
Comment #15
docans CreditAttribution: docans commentedI just want to know what i need to do after applying the patch so i can successfully migrate the field collection over. I dont see any documentation on what do do after applying the patch. Can anyone please point me in the right direction. Thanks
Comment #16
rozh CreditAttribution: rozh commented+1 on documentation. Could you provide some hints on how to perform migration.
Comment #17
mikelutzI'm working on this, and currently running into issues with multilingual setups. In particular, since field collection supported translation on the reference field and paragraphs doesn't, I can't properly map the content from one to another. I currently have a need to develop a means of migrating field collections from a multilingual d7 site over to d8 paragraphs, Which looks like it's going to depend on the patches in 2461695 to enable asymmetrical translations for paragraphs. I'll be posting my work over the next while, but I'm curious if anyone has any other thoughts on the issue. I will need to build something to meet my needs, but as far as a mergeable migration, it seems like we either need to not support multilingual field_collections with unlimited deltas or wait until asymmetrical translations for paragraphs are merged.
Comment #18
colanI think it's safe to assume #2461695: Support asymmetric translations will be getting in soon as it's green/RTBC, but we should state that it's a dependency here for i18n (until it gets merged). So let's get support in for that even though there's a small chance it might change somewhat.
The other option is to support it in a follow-up, but that feels like too many moving parts here.
Comment #19
heddnFor anyone stumbling on this issue, the patches earlier in this issue are good interim steps to run a field collection to paragraphs migration. But they probably won't get committed. The actual work for all this is happening down in the child issues. Therefore, hiding all files so as to avoid some confusion.
Comment #20
miro_dietiker@mikelutz Good to see progress with shipping the migration out of the box. I understand that this is a blocker then for field collection replacement. Let's cover this requirement in the experimental widget. I don't want to touch the classic widget functionality.
I'll redirect all efforts for this to happen into #2904705: Support asymmetric translation in experimental widget as a standalone issue.
Comment #21
mikelutzThis is the combined patch from all the child issues. It is passing tests, and seems to be working with the exception of translations.
Comment #22
mikelutzComment #23
miro_dietikerCan you please provide some more information about how committing of these would work?
Is the combined patch just for a combined testing and we keep committing the issues individually with individual test coverage?
Or do we need to commit the combined version?
Comment #24
mikelutz@miro_dietiker - It really depends on your preference. I have done development in the individual issues because they were there, and it did break the work down into bite-sized chunks that are easy to review. In truth, they have been a pain for me, as each issue depends on the previous one to work. I have to keep rebasing my local copies to try and keep all the patches straight, and while I could do some individual unit tests on each piece, full test coverage really requires them all to be combined. It's about to get worse when I tackle multi-lingual support today, which will touch all the issues at some level. Finally, any real world testing requires all the patches together, applied in the right order, and I figure no one will actually do that, which is why I posted the combined patch here, for anyone who actually can try to use it to migrate field collections to test.
My preference as a developer would be to shut the child issues down, and continue to work on D7 Field Collection and D7 Paragraphs migration in this thread off of the latest patch I've supplied, and commit one big patch when everything is ready. The only thing that might be helpful to separate out, if it could be committed quickly, would be the changes I've made to the fixture to support more advanced testing of things like revision, widget settings, and formatter settings. That would reduce the size and increase the readability of this patch and make things easier. If that's acceptable, I can make a separate issue for it, and add a patch for the fixture which passes the current tests, and then rebase this patch against it, and we can continue work in here.
Otherwise, I'll adjust my workflow to whatever you prefer. My main goal is to get the code available for people to use and test, and ultimatly get committed in whatever way is easiest.
Comment #25
miro_dietikerI'm only looking at the global test pass result while trusting on the review of migrate experts like heddn and Berdir.
It's still great to have the individual issues updated for specific reviews of the plugins for feedback.
In the best case they just confirm then that things are fine. We can always deal with improvements in follow-ups.
Comment #26
mikelutzThis new patch includes the fixture I am hoping to have committed from https://www.drupal.org/project/paragraphs/issues/2944844#comment-12482759 . That fixture has nested field_collections for testing, which exposed some bugs in the entity_reference_revisions destination plugin which are worked around in this update.
Comment #27
heddnHmm, I tried. I really did. But after getting to half-way, I really think we need to take things slower and step-by-step. The attached patch that combines all the things is really hard to review. Let's split things out to the sub issues again. And if that means we loose some momentum, so be it.
Below is the first part of my review. When addressing, please move the changes back into the separate issue again. And let's perhaps pend the follow-on issues until we get the first issue to land. That will mean less churn and re-work.
Can this label be more descriptive? Or perhaps make it singular to match the id?
Extra whitespace isn't needed.
Nit.
I think we want to retrieve the 0 and 1 position for paragraphs?
There's some work going on over in #2944597: Make the tests for plugins in plugins alter stricter that makes me think this should be a stricter check.
Now all of this is in a single issue, can we consider renaming this again? 'field_collection_strip_prefix' seems much more descriptive of what we are doing.
Let's call parent::getDerivativeDefinitions(). That will do the same thing, but also let the parent adjust things. I'm thinking of the deriver alter logic that will hopefully land someday soon.
Is this what core is doing too? Can we add a comment why we are catching and not doing anything w/ the exception?
This loop could use some comments on what is going on.
This looping logic could use some descriptions on what it is doing.
I think this needs to be pulled out of here into a separate issue. I don't think this will ever land in paragraphs. The logical place is over in ERR.
I think all of this needs to be pulled out of the field plugin and added to the migration yml itself. Meaning, add a process plugin that wraps all of this logic and passes in a list of migration names. Then we can abstract/hide all this complex 'target_id'/'target_revision_id'/'migration_lookup'/'extract' mess. And this process plugin should land in ERR, not in paragraphs.
This can extend static_map.
Usually we use the plugin id instead of the class name.
Comment #28
heddnFirst issue I'd like to see land is #2944844: Update testing database fixture to allow for testing more complex interactions, followed by #2911242: Create field collections source migrate plugin.
Comment #29
mikelutz@heddn, alright, I will reroll again piecemeal, and we can focus on getting each piece committed in smaller chunks in order. #2944844: Update testing database fixture to allow for testing more complex interactions was committed this morning, which is going to help a lot. I agree with this strategy, the next step is getting https://www.drupal.org/project/paragraphs/issues/2911242 RTBC and committed, along with #2911243: Create field collections field plugin which can be done concurrently. I will reroll those with dedicated unit tests against the new fixture such that they don't depend on each other. They won't provide any functionality on their own, but once they are committed, we can reopen
Ultimately I need this for a commissioned project pretty quickly, which is why I ran with everything. While I've now got a patch I can use for my project, I would very much like to see this work given back and eventually committed, although I will likely have less time to spend on it going forward.
That said, I have a few comments/questions about the above feedback items.
#4, 11, and 12 are all tied together and deal with what destination plugin(s) to use for the actual paragraph entity itself, and I spent quite a bit of time going back and forth on the issue. The entity destination only provides the entity id as an identifier, the entity_revision destination only provides the revision_id and with an entity_reference_revisions field, we need to be able to access both the entity_id and revision_id from a migration lookup. My problem was that the entity_reference_revision destination seems to be fundamentally broken in that it will not allow you to create a new revision of an existing entity, nor will it allow you to create stub entities. Prior to overriding the class, I submitted #2944836Migrate destination plugin attempts to update new entities, preventing creation of stub entities but truthfully, I can't figure out why that destination was written the way it was originally, and while I feel like my logic makes more sense and the original is buggy, with no existing test coverage around that class, I can't prove my solution won't affect someone else's existing migration. So, this is why I chose to extend it, either as a temporary measure until we can come to consensus around #2944836, or, if I can't convince ERR to change, it would still work, or we could extend off of EntityRevision directly and implement the rest of the ERR destination logic so we weren't depending on ERR. I did reference that ERR issue at the top of the destination plugin.
The fact of the matter is this is the destination for the paragraph entity itself, not the field which references it, and that isn't really ERR's job. ERR should be concerned with the field, not the entity which the field references. If ERR is going to be opinionated as to how that entity is saved in a migration, then it should support stubs and creating new revisions, I think.
As far as #12 goes, I'm up in the air. migration_lookup does such a good job of trying to find the reference, and creating a stub if it doesn't exist, I feel like any attempt to abstract that away in ERR is going to be far more messy, and unnecessary since the core plugins exist specifically to do that work, and it again hits the issue of ERR being opinionated about the entity it references rather than focusing on the field. That being said, I don't care where the logic goes as long as it works.
Finally, with issue #6 on the strip prefix plugin, I've actually come around to the thought that we should just get rid of it and just replace it with substr. When I first started working on this issue, one of the first patches I pulled to start working from had the 'prefix: field_' in the yml file, and the logic in the source to strip it, but I don't know any situation where field collection fields don't start with field_. I can't think of any situation where I need the logic of specifying or checking that. Everywhere it's used, I really just want to strip the first 6 characters, and the further I get in this the bigger of a PITA it is trying to pass around that value, which is essentially a constant. Can you think of any situation where someone might want to change that value in a migration alter or otherwise?
Comment #30
heddnRegarding ERR, that was just an oversight and copy/paste from what core does when that piece of code was written. I was pretty instrumental in its original implementation, there is test coverage for a first time creation. And I thought for a re-save too. But I could be wrong.
I'm +1 on using substr to replace. We have some precedent on doing that in other places and it seems fairly sane.
And want to say a HUGE thanks for all your work on this. I'd love to see it keep moving too. I've got some time to help move things along myself. So, on to #2911242: Create field collections source migrate plugin and #2911243: Create field collections field plugin
Comment #31
mikelutzDoh! You are right, there is a test class for that, it was buried and I missed it. Well, on the positive side, my change still passes the original test then, which helps my case. I just need to add a test for what I saw was broken, so that's good.
Comment #32
StevenPatzHey guys I too have a client who is doing a migration. D7 has Field Collections, D8 will have a paragraph to hold that info. So any help with testing etc you need I can help with.
I'm currently, after patching, trying to get the migration to work as described above.
Comment #33
mikelutz@StevenPatz That's awesome, I'm very interested in hearing how far you get with the latest patch.
Comment #34
StevenPatz@mikelutz Currently I am building out the PHP that maps FC columns to Paragraph field. Hoping to get that run locally tomorrow ( the FC we have is like 15 fields so it's a bit time consuming )
Comment #35
mikelutzThe latest patch here should do all that automatically, although it may not apply against the latest -dev because I had part of it committed yesterday.
Comment #36
StevenPatz@mikelutz Oh that's interesting. Let me test.
Comment #37
noel.delacruz CreditAttribution: noel.delacruz commentedHi, would like to ask which version of the module should I apply the patch to? I'm using the latest dev build and doing some test since I have a drupal 7 with field collections. When I tried running
drush ms
it only showsupgrade_d7_field_collection_type
. Is that supposed to be it or am I missing something?Comment #38
heddnYou asked 9 hours ago. At 5 hours ago, another piece of the pie #2911242: Create field collections source migrate plugin landed in the dev branch. Next up is #2911243: Create field collections field plugin. All of everything that is still remaining will need re-rolls (potentially complicated re-rolls). But the happy thing is that we are 2 or 3 issues from having full support for migrating paragraphs or field collections from D7 to D8.
Comment #39
mikelutzComment #40
mikelutzFor those interested, here's a reroll against the latest -dev. Work is being carried out in individual issue queues now, this includes some extra work on the content migration that isn't quite ready for commiting yet, but might be worth testing.
Comment #41
noel.delacruz CreditAttribution: noel.delacruz commentedI'm getting an
input should be an array
error when I tried migrating my nodes toDrupal 8
. Has anyone encountered this issue using the latest patch. Below is myYML
configurationComment #42
mikelutzThat actually looks like a migrate_upgrade bug. The code they run to add the upgrade_ prefix to the migration_lookups doesn't seem to set the keys for the source_ids., so your migration_lookup fails, and sends a scaler to the extract process, which expects an array. Adding the upgrade_ prefix for the keys there would help you, but we will need to file a bug with migrate_upgrate to get that fixed.
Comment #43
mikelutzAdding https://www.drupal.org/project/migrate_upgrade/issues/2946208 as related
Comment #44
StevenPatzI'll likely be trying out the latest -dev with patches this afternoon. Should I apply the latest patch here or individual issue patches ( those still in progress )
Comment #45
mikelutzGive #40 here a shot. it's got 2911243: Create field collections field plugin and an initial version of #2911244: Field collections derriver and base migration rolled in. It's the most up to date/farthest along code that I've got.
Comment #46
StevenPatz@mikelutz
Do I need to create Paragraph/add fields to it before running the migration? I have two yml files. One is called d7_field_collection_authors it maps source fields to destination fields. Doing a drush ms shows this after running it:
d7_field_collection_authors Idle 1701 1701 0 2018-02-20 16:07:31
I then modified my already built content type migration to do a migration_lookup of d7_field_collection_authors, this is what I see for drush ms:
d7_node_not_real_name Idle 304 297 0 2018-02-20 16:11:09
So it looks like the migrations are running okay. But when I spot check some of the not_real_name contents, I just see
"No paragraph added yet" and a button to add one.
I'm definitely on the -dev version and the patch from #40 applied cleanly.
Comment #47
StevenPatzJust checked the messages table for the not_real_name and the 7 or so that failed have this:
Comment #48
mikelutz@StephenPatz . The code was designed to run with the standard Drupal Migrate UI, so dependencies are a bit tricky if you are running custom migrations. It's tough without seeing the actual migrations you are working with, but the way I've been testing is to run migrations in the following order.
1) d7_field_collection_type from the module. This should create the actual paragraph type that cooresponds to the field collection
2) d7_field followed by d7_field_instance should load both the old field_collection field as a new entity_reference_revisions field targeting paragraphs, as well as load the field and field instances of the old field collection against the new paragraph created from 1. If you need to run a subset of these to just get the field collection fields, there are a bunch of processes from the paragraphs.module hook_migration_alter that you will need to add to make sure the field collection bundles translate over.
3) d7_view_modes, d7_field_instance_widget_settings, d7_field_formatter_settings. These are optional, but if you don't run them you will have to manually add those fields to the form mode and view mode.
4) d7_field_collection and d7_field_collection_revision to load all the new paragraphs. Those are derived into separate migrations for each field_collection, again, you can manually create ones for specific field_collections, just follow the format of the included ones and add the field_name to the source plugin.
5) Finally run d7_node, and if you are manually mapping the fields, the lookup process for the paragraph is a little complicated now, see Drupal/paragraphs/Plugin/migrate/field/FieldCollection::processFieldValues() to see how to set that up.
It does process from a migrate_lookup to an extraction, and your error message is something using the Extract process plugin without specifying the index, but all the migrations included in the patch definitely have a hard coded index array for each extract process, so I don't think that is coming from the patch.
Comment #49
noel.delacruz CreditAttribution: noel.delacruz commentedHello @StevenPatz, can you share with me your migration yml for the field collection and nodes you are trying to migrate. Currently doing multiple tests migrating/converting field collections to paragraphs
Comment #50
noel.delacruz CreditAttribution: noel.delacruz commentedHello @mikelutz, sorry which part in YML configuration should I add the upgrade_ prefix? Not familiar with the keys you are currently pointing.
I'm also currently trying your process. In Step #2 I'm getting this error when I try migrating my
upgrade_d7_field_instance
. The same error occurred when I try migratingupgrade_d7_field_instance_widget_settings
andupgrade_d7_field_formatter_settings
Here is my migration yml for the field instance.
Comment #51
mikelutzThat error suggests the d7_field_collection_type migration had a problem. Did you have a field collection named field_edia? Did you have any skips or problems with that migration?
Comment #52
noel.delacruz CreditAttribution: noel.delacruz commented@mikelutz, The field collection field_edia doesn't exist and I didn't have any problems when I migrated
upgrade_d7_field_collection_type
. I actually have only one field collection on my drupal 7. I'm actually testing this on a vanilla instance of Drupal 7 and 8Field Collection
https://imgur.com/hrB5KVg
Fields
https://imgur.com/Hm3D36R
Comment #53
mikelutzHmm, somehow it's stripping the first 6 characters off the bundle name twice. It should be changing field_multimedia to multimedia, but it's changing it to edia. I don't see why fof the top of my head, I'll have to look more closely in the morning. There should be tests against that.
Comment #54
noel.delacruz CreditAttribution: noel.delacruz commentedAside from that most of the fields successfully migrated. I guess it might take a while before we get patch that will allow us to migrate the content within th field collections.
Comment #55
StevenPatz@MikeLutz I'll post the migrations in a bit, have to clean out any identifying remarks etc. Also will follow the updated steps you posted, I know that I've run d7_field_collection_type and that two FCs come over ( only need one ) in the description of the paragraph type I see "Migrated from field_collection field_notice_author" I think I'm close. I'll have an update in a few.
Comment #56
StevenPatzThis is migrate_plus.migration.d7_field_collection_authors.yml
Comment #57
StevenPatzThis is the Content Type I am then trying to get the Migrated FC into:
Comment #58
mikelutz@noel.delacruz . The latest patch is supposed to moving content over if the migrations are done correctly. I'm still not sure why yours is stripping 12 characters from the bundle instead of 6, I can't seem to duplicate the behavior in the tests.
Comment #59
mikelutz@stevenPatz The field_notice_author process in your content type needs to return a target revision id in addition to a target id. (necessary for an entity_reference_revisions field, though I think that is actually a bug on the ERR side). If you aren't running a migration of revisions, and assuming only one copy of each paragraph, you could try:
Comment #60
StevenPatz@MikeLutz Busy day today with this and other things BUT. Steps 1-3 of #48 worked for me. I have a new Paragraph type with the fields and widget items imported in. Steps 4-5 soon, but I'm feeling good about this. Great work on the changes.
Comment #61
noel.delacruz CreditAttribution: noel.delacruz commented@mikelutz, just to make sure I'm in the same page as everyone. I'm using the dev build of paragraphs module. I tried applying the latest patch and I think it didn't patch cleanly
curl https://www.drupal.org/files/issues/2897021-40.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch | patch -p1
Comment #62
mikelutzHere's a reroll.
Comment #63
noel.delacruz CreditAttribution: noel.delacruz commented@mikelutz, thank you the patch applied cleanly. By the way you mentioned on how to fix my issue regarding
input should be an array
and that I should add an upgrade_ prefix in the keys in order to fix the issueIs this the part you mentioned that I should add the prefix in the
source_ids
section? Apologies for my ignorance since I'm still new with migrationComment #64
noel.delacruz CreditAttribution: noel.delacruz commentedJust did another test today trying to convert my field_collections to paragraphs was unsuccessful in migrating the content. I really feel I'm close to trying to get this to work since @mikelutz said the latest patch will be able to migrate the content also but unable to figure where the issue is coming from and how do I fix it. I'll share here what I have accomplished so far.
Drupal Version - 8.4.5
Using Drush I download the necessary modules -
Migrate Plus (8.x-4.0-beta2)
Migrate Tools (8.x-4.0-beta2)
Drupal Upgrade / Migrate Upgrade (8.x-3.0-rc3)
Paragraphs (8.x-1.2+47-dev) with the latest patch #62
And then followed @mikeLutz steps in migrating the items in the following order
1.) Migrate
upgrade_d7_field_collection_type
. Didn't have any problems here2.) Migrating the
upgrade_d7_field
was also successful but when I tried migrating the field instance(upgrade_d7_field_instance)
I came across an error which @mikelutz cannot replicate. I'm using the latest patch here by the way. The error below also show itself when I tried migratingupgrade_d7_field_instance_widget_settings
andupgrade_d7_field_instance_widget_settings
. I only have one field collection registered on my Drupal 7Field Collection - https://imgur.com/rtx4yaY
3.) Migrating
upgrade_d7_view_modes was also clean
,upgrade_d7_field_instance_widget_settings
andupgrade_d7_field_instance_widget_settings
had issue which is the same as above4.) Before migrating my field collections I did a comparison with @StevenPatz YML configuration. I noticed some difference. Here is my YML configuration. On his YML configuration, the field plugin are set to
iterator
but mine is set toget
, and the destination plugin is set toentity_reference_revisions:paragraph
while my configuration is set toparagraph
. Decided to follow his configuration hoping it might work. I also did the same for myupgrade_d7_field_collection_revisions_authors
5.) Before migrating my content, I also did a comparison with @StevenPatz.
Below is my YML configuration. I noticed that my machine name is incomplete. It only shows
upgrade_d7_field_collection
, but shouldn't it beupgrade_d7_field_collection_authors
which should be correct? Because that is the machine name of my field collectionupgrade_d7_field_collection_authors
.6.) After making the modifications and changes and migrated the content type/node, the fields are empty as shown in this screenshot link below.
https://imgur.com/pNkYFTc
SQL Table:
https://imgur.com/JindD4r
Already tried creating a fresh drupal instance of 7 and 8. Already also tried re-installing the modules and applying the patch. But I still get the same result
Comment #65
mikelutz@noel.delacruz
You are getting close. Still not sure on your error migrating instances, but if in d8, you have author and email field attached to your new paragraph type, then it is working. There may be an incompatibility with some other migration I'll need to look into, but I have a few revisions left to make on the field plugins so hopefully I'll find it with further testing. I did not do any testing with migrate_upgrade, I've been using drupal console's migrate:setup, and the migrate test classes for testing. migrate_upgrade does a bunch of altering of the migrations, which might be causing problems, such as the source_ids issue (yes, that is where you need to change the keys btw). I do need this to work with migrate_upgrade, so I want to identify any migrate_upgrade issues, but I'll need to work out whether they are bugs in migrate_upgrade or this code. I'll give it a shot with migrate_upgrade and see what happens.
You *shouldn't* need to change your machine name to add the _authors, since that is a derivative of the d7_field_collection migration, it should be pulled in automatically.
The 'paragraphs' destination plugin is one I wrote to work around a couple bugs in the entity_reference_revision:paragraphs destination plugin related to revisions and nested field collections. if you aren't worried about revisions and don't have nested field_collections, then ERR:paragraphs should be okay, but if you are doing a separate revisions migration, use the paragraphs destination, it will handle things correctly until ERR's bug is resolved.
Either get or iterator (now sub_process) should work for fields. If yours was generated by migrate_upgrade with get, then that should be fine.
So, at first glance, yours looks like it should have worked.
What does your paragraphs_item table look like? Did it create the entities but not fill them, or did it not create them at all?
Comment #66
StevenPatzUpdates, and some Progress.
So I decided to just run the built-in migrations so I ran d7_field_collection_revisions:notice_author etc instead of the custom yml, thinking if I can at least get the data into the D8 system, cleaning up would be easier. I then changed the custom Content Type migration to reference the built-in migrations:
This is drush ms:
All looks okay ( the notice_author didn't fully do because the FAX number from D7 is plain text with letters and words in and in D8 I'm using telephone, but enough did migrate for testing )
I created a View to see what data is in the Paragraphs and I get a few pages of actual migrated content from D7. But when I spot check xxxx_notice nodes, I can't find any that have a paragraph attached like they should. I feel like I'm really close, but can't tell if it's still some code that needs added/fixed in the module or if it's just not the right yml settings for the migration. For the 12 that didn't migrate drush mmsg shows:
Should I apply the latest patch? Any other ideas/tips?
Comment #67
mikelutz@StephenPatz You are getting really close. I think your extract error messages are due to #2890690: MigrateLookup plugin has inconsistent return values
Basically the paragraphs that are failing are causing a stub creation, which is returning only an entity_id to extract. So these errors are for the content items referencing the paragraph items that are missing anyway.
Honestly everything looks right, except in your content type process you mapped to 'notice_author' instead of 'field_notice_author'. That looks to have changed from what you posted previously, so I'm not sure if that made it into your migration, or was just an error in copying it here.
Comment #68
StevenPatz[EDIT} Hmmm, though the machine name in the content type is field_notice_author, the Paragraph machine name is notice_author and the source machine name is field_notice_author. Let me re-run these again]
Yeah I doubled checked machine names this morning and notice_author is the name of the Paragraph, that was a previous typo and had caused a lot more of the nodes to not be migrated. Definitely need to get this working for our migration, so I'll update as I keep playing around.
Comment #69
rozh CreditAttribution: rozh commentedmikelutz, You're amazing! Great job here.
I tried latest dev of Paragraphs with patch from #62. I used regular migration with core Migrate UI module but haven't succed.
Paragraph types and fields was created but content wasn't. Actually nodes with at least one field collection item wasn't migrated.
Comment #70
mikelutz@StephenPatz yeah, it's a little complicated. the field name is 'field_notice_author', which is an entity reference revisions field, which references paragraph entities, and allows paragraphs of bundle type 'notice_author'. For field collections, the field name and bundle type were always the same string with a one to one relationship, but they are completely separate concepts in paragraphs.
@rozh I'm still working on the content migrations, I've gotten some to come over, but there are still issues being reported, so there is still work to do. Keep an eye here for updates. We are really pushing to have https://www.drupal.org/project/paragraphs/issues/2911243 reviewed and committed before I spend too much more time working on the content migrations.
Comment #71
StevenPatz@MikeLutz in re #70 is there a separate issue I can look at/review/help with regarding content migrations? it seems like you're saying what I've experienced, Paragraph type gets created, fields in that type get created, and the data from FC is loaded, but....getting that Paragraph attached to a node is not working?
Comment #72
noel.delacruz CreditAttribution: noel.delacruz commented@mikelutz
Is there any difference with using drupal console's migrate:setup? I might want to try it just to get this working. Do you have any documentation on how I can use Drupal console's migrate. Do I also need any other migration module or dependencies?
Comment #73
mikelutz@StevenPatz, I really need#2911243 reviewed and committed before I can put serious work into the content migrations. That is the issue that migrates the fields and field instances, and any content migration work depends on it. The patch here contains an earlier version of that patch along with some preliminary content work I did.
@noel.delacruz drupal consoles migrate:setup just sets the migrations up differently, doesn't try to change the migration ids and such. I'm not saying it's better or worse, just a different method. Good news on your side though, @heddn did post some steps to replicate in that thread I mentioned above that seem to trigger your field instance error, so I should be able to track it down soon.
Comment #74
colan@mikelutz: This should help with referring to other issues, from the More information about text formats link, so that you don't have to add the links yourself:
#2911243: Create field collections field plugin, for example.
Comment #75
StevenPatz@mikelutz I've applied the latest patch from https://www.drupal.org/project/paragraphs/issues/2911243 and see no errors ( still can't get the actual nodes with their paragraphs data). I'll be working on this over the weekend, I'll check back here to see if there's any progress.
Comment #76
noel.delacruz CreditAttribution: noel.delacruz commented@mikelutz
Drupal Console migrate doesn't generate the YML file for migration automatically right? Unlike Migrate Upgrade which can generate the configuration via drush
durhs migrate-upgrade --configure-only
and then export it usingdrush config-export --destination="path"
.And If I'm going to drupal console migrate I need to also build my own module to migrate those item I have right?
-------------------------------------------------------------
Tried using drupal core migrate module, and I get the error
array index missing, extraction failed
when migrating my nodehttps://imgur.com/MHLsDjl
I believe the same issue happens if I try drush migrate-upgrade. These are nodes with field collections that are embedded.
drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven
The results differ if I tried using a different command on drush migrate-upgrade
If I tried following @mikelutz steps indicated above, I'm still getting
input should be an array
despite installing the latest release of migrate upgrade that addresses the issue on the source ids (Migration Lookup source_ids array keys are not properly prefixed when creating migrations.
). Below is the YML file generated by migrate upgradeComment #77
noel.delacruz CreditAttribution: noel.delacruz commentedOk, let me compile/summarize the test I have done so far in trying to migrate/convert the field collection to paragraphs using the available resources/modules and patches
Drupal - 8.4.5
Drupal - 7.56
Modules
1.) Migrate Upgrade (8.x-3.x-dev)
2.) Migrate Tools (8.x-4.0-beta3)
3.) Migrate Plus (8.x-4.0-beta3)
4.) Paragraphs (8.x-1.x-dev) with latest patch applied
Test 1
I tried migrating the content and fields using the following steps below via drush
1.) drush migrate-upgrade --configure-only
2.) drush mim -y --all
I'm still getting the error Missing bundle entity when migrating
field_instance
,field_formatter_settings
,field_instance_widget_settings
.I noticed that the node that contains field collection failed to migrate and rendered a migration message
Input should be an array.
. Below is the screenshot that contains the error within the SQLhttps://imgur.com/f85q7V5
The
paragraphs_item
table were filled as shown in this screenshot.https://imgur.com/48blUov
Data was not migrated successfully as shown in the screenshot. As you can the tables were empty.
Field Authors Name - https://imgur.com/YLeTJBR
Field Email - https://imgur.com/hyVT8kc
The fields within the field collection was migrated but did not attach to the paragraph types that was created. You have to attach those fields manaully, screenshot below
https://imgur.com/yO64RJu
----------------------------------------
Test 2
For the second test, I used a single command to migrate or upgrade all field and entities from Drupal 7 to Drupal 8
drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven
and it had a different resultThe node or content type that contains the field collection was not migrated successfully and had a different error compared to the first test. It showed an error "Array index missing, extraction failed". You can see in the screenshot below.
https://imgur.com/SVW3Q0f
Contents or Data within the fields from the field collection was reflected on the paragraphs field table. As you can it has the same data as the node in drupal 7
Author Email Field - https://imgur.com/Vw3DNko
Author Name Field - https://imgur.com/ktnq2Ed
Paragraphs Item Table - https://imgur.com/Th5VU7I
Paragraphs Item Field Data Table - https://imgur.com/XMLaLUt
Drupal 7 Node - https://imgur.com/XzWXD03
The Field have also been attached to the Paragraph Types
https://imgur.com/J2annAr
----------------------------------------
Test 3
For the third test, what I did was to configure or modify the YML configuration for the node in order
Input is an array
and array index missing. Exported the configuration
drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven
Edit the node YML configuration that has paragraphs/field collection config
Remove this part within the configuration
The result was it managed to migrate the node but the data from the field collection converted to paragraphs did not attach to the node.
Not sure what else I need to do in order to make this work since I have D7 Project that contains field collection. If you have any more ideas or suggestion that I please share it with me so I can do some tests. I tried using Drupal Console migrate:setup, but after the part of entering your local directory
Local file directory containing your source site
nothing happened did I miss anything?Drupal Console migrate:setup - https://imgur.com/4BkbBxh
Drupal Console migrate:execute - https://imgur.com/7dq2Y6i
Comment #78
mikelutz@noel.delacruz Wow!, thank you so much for the detailed data. I am back looking into this today to track down some of the issues you and StephenPatz have been having. Looking at the error messages, I suspect we are running into issues with creating paragraph stubs and properly updating them, and I still think migrate_upgrade is doing something as well, but between your report and one provided by @heddn in #2911243: Create field collections field plugin I should have enough information to duplicate the issues in the debugger and see what's going on. Stay tuned.
Comment #79
mikelutzHere's a new patch for you guys to chew on. It should solve the issue with field instance errors, though I'm not really happy with how it does it at the moment. You would need to start fresh and rerun migrate-upgrade for it to work.
Comment #80
StevenPatz@mikelutz Hoping to apply patch later today or first thing in the morning.
Comment #81
mikelutzFixing my broken tests
Comment #82
StevenPatz@mikelutz That last patch did it. I have my Content Type with a Paragraph attached with the actual data migrated from FC to Paragraph. My end is not nearly done as I'll need to make a custom migration ( was using all built-ins to help here and to prove that it could be done) But your work ( as well as others) is coming along quite well.
Comment #83
noel.delacruz CreditAttribution: noel.delacruz commentedI'm getting
Input should be an array
when I tried migrating my node that containsfield_collection
config. I'm using the latest release ofmigrate upgrade
Comment #84
mikelutz@noel.delacruz Are you sure the actual field collection migration is running first? There is an issue with the stub creation returning non-arrays to extract. I'll have it worked out in the final migrations, but for now, make sure the field collection migration runs first.
Comment #85
noel.delacruz CreditAttribution: noel.delacruz commented@mikelutz
Yes, these are the steps that I have taken
1.) Drush migrate-upgrade --configure-only
2.) Import Migrate Configurations drush cex
3.) Drush Migrate Import the items in the following order:
-- upgrade_d7_field_collection_type
-- upgrade_d7_user_role
-- upgrade_d7_user
-- upgrade_d7_field
-- upgrade_d7_field_instance
-- upgrade_d7_view_modes
-- upgrade_d7_field_formatter_settings
-- upgrade_d7_field_instance_widget_settings
-- upgrade_d7_field_collection_authors
-- upgrade_d7_field_collection_revisions_authors
-- upgrade_d7_node_press_release
If I try migrating using a single command from migrate-upgrade this is the error that I receive
Array index missing, extraction failed.
drush migrate-upgrade --legacy-db-url=mysql://root:@127.0.0.1/foundationseven
By the way how long will it take in order to sort the
array issue
? I know there is a time difference so I'll try to get up as early as I can for testing. Really need this, so far field_instance have been migrated cleanly. Thank you for that..Comment #86
mikelutzre-rolling this patch to be inline with #2911243: Create field collections field plugin
Comment #87
StevenPatz@mikelutz
Thanks for keeping this issues patches up to date. This latest applied cleanly for me, and I am in the process of making a custom migration with all of this work ( because we don't need a lot of the fields the built in migrations do.)
I'll update as I move forward.
Comment #88
noel.delacruz CreditAttribution: noel.delacruz commentedWill try the latest patch and use and clean drupal instance. Going to use migrate-upgrade since it is the only tool I know how to use for migration
Comment #89
noel.delacruz CreditAttribution: noel.delacruz commentedApplied the new patch and I'm getting some new notifications.
curl https://www.drupal.org/files/issues/2897021-86.paragraphs.META-Migrate-support-for-importing-field-collections-as-paragraphs.patch | patch -p1
------------------
Comment #90
mikelutz#2911243: Create field collections field plugin was committed a couple hours ago which contains part of this patch, this patch will need a complete re-roll now. Fortunately that's the last blocker before I can work on the content import part, so I can get started on that again tomorrow.
Comment #91
noel.delacruz CreditAttribution: noel.delacruz commentedJust a quick update, I followed @mikelutz steps mentioned in migration so far these are the results.
I'm using migrate-upgrade by the way. Fetch the migration the configuration first before starting.
drush migrate-upgrade --configure-only
1.) Migrated first upgrade_d7_field_collection_type - Migration was clean
2.) Migrated some of the dependencies and requirements before proceeding to migrating the fields. These d7_user_role, d7_user, d7_comment_type, d7_node_type, d7_taxonomy_vocabulary
3.) Migrated d7_fields - Migration was clean
4.) Proceeded to migrate d7_field_instance, d7_view_modes, d7_field_instance_widget_settings, d7_field_formatter_settings. So far everything was clean
5.) Migrated the collection fields d7_field_collection_authors, d7_field_collection_revisions_authors. Migration was clean
6.) Lastly d7_node_press_release which is basically the node. There was no errors during the migration and I didn't get the
Input should be an array
errorUpon checking the data within the fields have been migrated on the SQL table as shown in the screenshot below
Authors Field - https://imgur.com/dDRo9Mx
Email Field - https://imgur.com/kZ5utbt
But upon checking the node. The fields were empty
https://imgur.com/3MjpPo9
https://imgur.com/3MjpPo9
Just a note, I created a new drupal 8 instance using 8.4.5. Downloaded the migration modules and applied the patch for the paragraphs-dev build. Also cleared the cache
Comment #92
mikelutzNow that the blockers are in, I'm moving the remaining work to the appropriate child issue - #2911244: Field collections deriver and base migration. I'm going to re-roll what's left of this patch there and keep working.
Comment #93
mikelutzComment #94
noel.delacruz CreditAttribution: noel.delacruz commentedHi @mikelutz, do I need to apply first the patch here then apply the patch from #2911244: Field collections derriver and base migration?
Comment #95
mikelutzJust use #2911244: Field collections deriver and base migration on the latest -dev now. Everything else has been committed.
Comment #96
miro_dietikerThere's a new 8.x-1.5 ERR release now so hopefully this should unlock progress in further Migration improvements.
Comment #97
davidwhthomas CreditAttribution: davidwhthomas commentedGreat to see the updates in this area.
I just want to note, I'm doing quite a bit of custom migration work and found I could use the following migration YML files to migrate the paragraph data, without the migration lookups. I deleted all the paragraphs data first with drupal console:
drupal entity:delete paragraph --all
and migrated with the same ID values as the d7 source DB.This includes migration of the field collection
item_id
to the paragraphid
and the reference to the paragraph on the host entity field.Paragraph item migration
Host entity migration
These migration configs can be imported via the config single import UI, or from the module
migrations
folder. They can then be run with drush:Maybe useful for others in a similar situation.
In the meantime, keep up the great work!
Comment #98
Jumoke CreditAttribution: Jumoke commentedWhat's the correct format for migrating fc into a paragraph on a node?
This works but I am now noticing that the results are widely inconsistent, sometimes the data simply doesn't make it in. And in cases where I have multiple FC fields on a nodetype, it's assigning a FC into the wrong Paragraph! Scary!
This format returns a "value" is not a valid entity message
Same as below:
My FC Migration into Paragraph Type looks like this:
Comment #99
mikelutzA side effect of the ERR destination, (and maybe one we should change) is that if it's a new entity, it enforces a new revision. So in your initial migration, you are keeping your ids, but your revision ids are probably changing, and need to be looked up. Check your database and see if the actual revision_ids match or not. I expect they won't. Try something like:
Comment #100
Jumoke CreditAttribution: Jumoke commentedThank you Mike for your help. I have some progress with the format below as suggested. Only thing now is that it brings in old revisions.
FC Migration
Node Migration
You'd notice I have the revisions part commented out in the FC migration. With or without it, it's currently bringing in the wrong revisions. What else am I missing here? Looks like it's picking up random revisions.
Comment #101
mirsoft CreditAttribution: mirsoft commentedI posted here a working solution to attach a parent entity to any existing paragraph or field collection that works for me (latest Drupal 8.6.2) with an assumption that I don't need to migrate the old paragraphs / field collection revisions, just attach the latest revision: https://www.drupal.org/project/entity_reference_revisions/issues/2990527...
Maybe it will help.
Comment #102
crash_springfield CreditAttribution: crash_springfield commentedWhat's the current status of this? Applying one patch I still get `The "field_collection_item" plugin does not exist.` while applying another patch I get `Plugin (d7_paragraphs) deriver Drupal\paragraphs\Plugin\migrate\D7ParagraphsItemDeriver does not exist`.
Comment #103
agudivad CreditAttribution: agudivad commentedDoes anyone found working patch?
Comment #104
juampynr CreditAttribution: juampynr at Lullabot for NBCUniversal commentedI did a successful migration using #2911244: Field collections deriver and base migration.
Comment #105
sukh.singh CreditAttribution: sukh.singh as a volunteer commented@juampynr it will be great if you can in detail mention how did you migrate field collections from D7 to D8?
Comment #106
cgalli CreditAttribution: cgalli as a volunteer commentedGreat work! It is a bit confusion for the non involved, though....
Could anybody please post examples of the needed config files to get a field_collection (D7) to a paragraph field (D8). That would be very helpful...
Many thanks
Comment #107
juampynr CreditAttribution: juampynr at Lullabot for NBCUniversal commentedSure @sukh.singh! I am on holiday now but next week I will add some notes here. Here is a hint though:
I installed the dev version of paragraphs, patched it, and then I used migrate upgrade and migrate drupal to generate migration entities, then I ran config migrations, which created paragraph types for each field collection, and finally I ran content migrations, which migrated content from field collections to paragraphs.
I posted an article with some of the commands involved at https://www.lullabot.com/articles/overview-migrating-drupal-sites-8.
Comment #108
bob.hinrichs CreditAttribution: bob.hinrichs commentedI ran into a migration error during field_formatter_settings import.. "The "field_collection_fields" plugin does not exist." This is in a display setting for the node type, when one selects "Fields Only" as the display for the field collection. I don't know if there is an applicable value that could be mapped for that formatter?
Here was my solution that prevents the migration from halting. In the configuration for d7_field_instance_per_view_mode, I added under the options/type map, under the values for field_collection:
field_collection_fields: entity_reference_entity_view
Comment #109
bob.hinrichs CreditAttribution: bob.hinrichs commentedI am using the recommended patch for Drupal 8.7 and I have hit a wall with it because of this little issue:
We are changing the machine name of the field in the node, that holds the D7 field collection that is becoming a paragraph on D8. This is done by changing the name via the yml configuration (uses string replace or static map plugin, etc.). In the case of the field name, this has created this problem:
1) The field machine name gets changed for the destination, when it is importing the field.
2) In FieldCollectionFieldInstanceSettings, the target bundle to be compared against is using row->getSourceProperty('field_name') as the target. However in our case the target's field name is changed, and the source no longer is accurate for this comparison. It should be comparing against the destination's not the source's values for the field name.
Going further, I encountered another, different problem.
I tried to fix this by overriding the class and setting my class's ID to use for field instance settings in my yml. This does not work though because the ids are hard-coded in FieldCollection class, in alterFieldMigration methods. I should be able to configure the plugins to use in the yml.
Comment #110
Sseto CreditAttribution: Sseto commented@Juampy. I was wondering if you ever had a chance to write an article on migrating field collections from D7 to D8? Thanks!
Comment #111
juampynr CreditAttribution: juampynr at Lullabot for NBCUniversal commented@Sseto hi! I followed this process, where I patched Field Collection module.
Comment #112
Sseto CreditAttribution: Sseto commentedThanks @juampynr!
Comment #113
j.kop CreditAttribution: j.kop commented@juampynr Hi! Is it also possible to migrate field collections inside D8? We currently use field collections 8.x-1.0-alpha1 and need to migrate a huge load of field collections to paragraphs.
I am not sure about how this differs from your nicely described D7 to D8 migration process as I do not know about the differences between D7 and D8. Could you give me a hint?
Comment #114
zoraxI try to migrate field_collection d7 to paragraph d8.
I use this patch.
When I migrate upgrade_d7_field_formatter_settings, I have error messages :
[error] The "field_collection_fields" plugin does not exist. like #108
In the configuration for d7_field_instance_per_view_mode, I added under the options/type map, under the values for field_collection:
field_collection_fields: entity_reference_entity_view.
Now errors messages disappears, but field_collection are not imported....
Is there someting I miss ?
1 - patch paragraph with #121
2 - modify d7_field_formatter_settings
3 - Do I need someting else ?
Comment #115
iuliad CreditAttribution: iuliad commentedHello,
Could I use this migration to migrate a single field from my field collection (which has only 1 entry per node, no multiple values), over to a single field on my node? Eg. my field collection has 1 field, a link, and now I have a single link field on my node and I would like to copy the data from the field collection directly into that field, without migrating the field collection into a paragraph. Thank you in advance.
Comment #116
DamienMcKenna@iuliad: You could use the code in this patch to help you work out how but it won't do that sort of data structure change for you.
Comment #117
juampynr CreditAttribution: juampynr at Lullabot for NBCUniversal commented@j.kop: for Field Collections to Paragraphs, see #2911244: Field collections deriver and base migration
Comment #118
iuliad CreditAttribution: iuliad commentedHello, I have managed to do this in the following way:
Migrate data from a content-type with one field collection attaches that has single values (no multiple values) to normal, single-value fields on a content-type:
1. Create a custom source plugin that adds the field collection's field values to the node source
2. Edit the content type's migration file to include the field collection field name and field values:
In this example, field_link is now a simple field on the new Drupal 8 node, and we will migrate the data from the Drupal 7 field collection's field field_link.
Comment #119
kevinquillen CreditAttribution: kevinquillen commentedJust ran an inline migration from D7 > D9. I can see that field collections were recreated as Paragraphs - however:
I read through the thread, but where can I start with getting data migrated in after I ran an inline upgrade?
Comment #120
jasonyarringtonFollowing. I am in the same situation as #119
Comment #121
iancawthorne CreditAttribution: iancawthorne commented@kevinquillen and @jasonyarrington have a look at issue https://www.drupal.org/project/paragraphs/issues/3145755 patch #28 and also have a read of my comment at #34.
Comment #122
livomex533 CreditAttribution: livomex533 commentedI came from a child thread of this here https://www.drupal.org/project/paragraphs/issues/3020433
Are there any examples for a D8 Field Collection > D8 Paragraphs module migration?
Comment #123
lrocha CreditAttribution: lrocha commented+1 on #122 comment
Comment #124
Wim LeersRE all the comments wrt field collections: see #3020433-12: Migration from D8 Field Collection to D8 Paragraphs.