Problem/Motivation

We need a method to upgrade/migrate from D7 field collections to D8 paragraphs.

Proposed resolution

Remaining tasks

a field collections source entity plugin, look at node source

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#30 interdiff.2911242.27-30.txt5.26 KBmikelutz
#30 2911242-30.paragraphs.Create-field-collections-source-migrate-plugin.patch34.9 KBmikelutz
#27 interdiff.2911242.24-27.txt6.14 KBmikelutz
#27 2911242-27.paragraphs.Create-field-collections-source-migrate-plugin.patch35.33 KBmikelutz
#24 interdiff.2911242.21-24.txt11.24 KBmikelutz
#24 2911242-24.paragraphs.Create-field-collections-source-migrate-plugin.patch33.26 KBmikelutz
#21 2911242-21.paragraphs.Create-field-collections-source-migrate-plugin.patch27.38 KBmikelutz
#21 interdiff.2911242.19-21.txt1.83 KBmikelutz
#19 interdiff.2911242.17-19.txt532 bytesmikelutz
#19 2911242-19.paragraphs.Create-field-collections-source-migrate-plugin.patch27.87 KBmikelutz
#17 interdiff.2911242.15-17.txt8.39 KBmikelutz
#17 2911242-17.paragraphs.Create-field-collections-source-migrate-plugin.patch27.84 KBmikelutz
#15 interdiff.2911242.14-15.txt4.45 KBmikelutz
#15 2911242-15.paragraphs.Create-field-collections-source-migrate-plugin.patch24.82 KBmikelutz
#14 interdiff.2911242.8-14.txt19.72 KBmikelutz
#14 2911242-14.paragraphs.Create-field-collections-source-migrate-plugin.patch24.33 KBmikelutz
#8 interdiff.2911242.7-8.txt1.83 KBmikelutz
#8 2911242-8.paragraphs.Create-field-collections-source-migrate-plugin.patch8.33 KBmikelutz
#7 interdiff.2911242.5-7.txt566 bytesmikelutz
#7 2911242-7.paragraphs.Create-field-collections-source-migrate-plugin.patch6.5 KBmikelutz
#5 interdiff.2911242.3-5.txt8.85 KBmikelutz
#5 2911242-5.paragraphs.Create-field-collections-source-migrate-plugin.patch6.5 KBmikelutz
#3 paragraphs-create-field-collection-item-migrate-source-2911242-d8.patch2.35 KBcitlacom

Comments

heddn created an issue. See original summary.

citlacom’s picture

Today needed to work on a migration of field collection items to paragraphs and after experiment few different approaches I found in issues, concluded that the patch from https://www.drupal.org/project/field_collection/issues/2757989#comment-1... was the most close to my expectations so did some coding styling tweaks and ported that patch to the paragraphs module and it worked fine, the migration of some field collection items to paragraphs was a success so I share the final result here.

The migration config looks:

id: field_collection_amenities
migration_tags:
  - 'Drupal 7'
migration_group: content
label: 'Migrate D7 amenities field collection to D8 paragraph entity.'
source:
  // This defines the source plugin attached in the patch.
  plugin: d7_field_collection_item
  // This is the field name of the field collection.
  field_name: field_amenities
process:
  type:
    plugin: default_value
    // NOTE: this is the paragraph type of destination.
    default_value: amenity
  langcode:
    plugin: default_value
    default_value: en
  default_langcode:
    plugin: default_value
    default_value: 1
  field_description:
    -
      plugin: skip_on_empty
      method: process
      source: field_description
    -
      plugin: substr
      source: field_description/0/value
      start: 0
      length: 512
destination:
  plugin: 'entity:paragraph'
migration_dependencies:
citlacom’s picture

heddn’s picture

mikelutz’s picture

Status: Closed (duplicate) » Needs review
StatusFileSize
new6.5 KB
new8.85 KB

Reopening this, as the field collection source is different than the field collection type source plugins.

I'm attaching my patch files which include 4 source plugins for field collections, field collection revisions, paragraphs and paragraphs revisions.

When it is time to include translations, I will likely update these to get more metadata from the node which references them to pass in.

No tests, but these files will be required for the other issues in this meta.

Status: Needs review » Needs work
mikelutz’s picture

Fixing a typo.

mikelutz’s picture

I've added a couple of tweaks here to the type plugins that are needed in later stages.

mikelutz’s picture

Status: Needs work » Needs review
miro_dietiker’s picture

I see multiple issues building on top of this.
As this has no test coverage on its own, should we really commit it separately?

mikelutz’s picture

@miro_dietiker I would probably not commit this on it's own. It's been rolled into the patch I posted on the meta issue already, and that has some test coverage. While these source plugins are self-contained, they do not do anything without the other patches.

miro_dietiker’s picture

The fixture is in, triggered a retest here.
@heddn requested to get this in separately. Are we thus moving some tests over?

mikelutz’s picture

I will rebase against the new fixture and add some test coverage specifically for this issue today.

mikelutz’s picture

Okay, This patch adds test coverage for each of the 6 source plugins. There is some minor tweaking of the type plugins already committed. I renamed Drupal\paragraphs\Plugin\migrate\source\d7\ParagraphType to ParagraphsType to match the plugin id, and maintain consistency. I refactored FieldCollectionType to not look for a prefix in configuration and just strip the first 6 characters when creating bundle names and descriptions. I unified the class descriptions across all 6 classes.

Pending review, I think this is ready to be committed and used for future migration work in the other issues.

mikelutz’s picture

I need to do better self review before I post.. I found a few places where code spacing or comments wasn't consistent between similar files, so I fixed them up. No functional changes here from #14, except changing the source id for the field_collection_type from the database numeric id to the field_name, to be consistent with the paragraphs type source.

heddn’s picture

Status: Needs review » Needs work

Lots of small nits. This is looking really good.

  1. similarity index 100%
    rename from migrations/d7_paragraph_type.yml
    
    rename from migrations/d7_paragraph_type.yml
    rename to migrations/d7_paragraphs_type.yml
    

    Renaming is fine as these are just place in time copies. No BC layer needed.

  2. +++ b/src/Plugin/migrate/source/d7/FieldCollectionItem.php
    @@ -0,0 +1,88 @@
    +    // Select node in its last revision.
    

    Copy/pasta. Probably can be removed.

  3. +++ b/src/Plugin/migrate/source/d7/FieldCollectionItem.php
    @@ -0,0 +1,88 @@
    +    if (isset($this->configuration['field_name'])) {
    +      $query->condition('f.field_name', $this->configuration['field_name']);
    

    It would be super helpful if we could add some doxygen comment to the class explaining what this config option is all about. It isn't just clear.

  4. +++ b/src/Plugin/migrate/source/d7/FieldCollectionItem.php
    @@ -0,0 +1,88 @@
    +    $bundle = substr($bundle, 6);
    

    Let's add a comment on what we are doing here. Or better yet, create a class constant with comments what this magic number is all about.

  5. +++ b/src/Plugin/migrate/source/d7/FieldCollectionItemRevision.php
    @@ -0,0 +1,31 @@
    +  public function getIds() {
    

    Is this just the revision id or is it revision id + item_id?

  6. +++ b/src/Plugin/migrate/source/d7/FieldCollectionType.php
    @@ -33,22 +33,20 @@ class FieldCollectionType extends DrupalSqlBase {
    +    // Remove field_ prefix for new bundle.
    

    Great comment. Better if we make this a class constant.

  7. +++ b/src/Plugin/migrate/source/d7/FieldCollectionType.php
    @@ -33,22 +33,20 @@ class FieldCollectionType extends DrupalSqlBase {
    +    if (isset($this->configuration['addDescription'])  && $this->configuration['addDescription']) {
    

    It would be nice to add some docs on what this config option is all about.

  8. +++ b/src/Plugin/migrate/source/d7/ParagraphsItem.php
    @@ -0,0 +1,86 @@
    +  public function getIds() {
    +    $ids = [
    +      'item_id' => [
    
    +++ b/src/Plugin/migrate/source/d7/ParagraphsItemRevision.php
    @@ -0,0 +1,31 @@
    +  public function getIds() {
    +    $ids = [
    +      'revision_id' => [
    

    I think that the data model in D7, paragraphs used a composite ID as well. If I'm correct, then this should be a combo of both values.

  9. +++ b/src/Plugin/migrate/source/d7/ParagraphsItemRevision.php
    similarity index 86%
    rename from src/Plugin/migrate/source/d7/ParagraphType.php
    
    rename from src/Plugin/migrate/source/d7/ParagraphType.php
    rename to src/Plugin/migrate/source/d7/ParagraphsType.php
    

    Can we keep a BC layer and the former class in the code base? It should extend this new class and add a trigger error, etc as listed in https://www.drupal.org/core/deprecation#how-class. Its been in the code base for a while now, someone might be using it. And its super easy to add the BC layer, so let's just do the right thing.

  10. +++ b/src/Plugin/migrate/source/d7/ParagraphsType.php
    @@ -30,7 +30,7 @@ class ParagraphType extends DrupalSqlBase {
    +    if (isset($this->configuration['addDescription'])  && $this->configuration['addDescription']) {
    

    I think a !empty check would do mostly the same thing. No?

  11. +++ b/tests/src/Kernel/migrate/FieldCollectionItemRevisionSourceTest.php
    @@ -0,0 +1,44 @@
    + * @group paragraphs
    
    +++ b/tests/src/Kernel/migrate/FieldCollectionItemSourceTest.php
    @@ -0,0 +1,57 @@
    + * @group paragraphs
    
    +++ b/tests/src/Kernel/migrate/FieldCollectionTypeSourceTest.php
    @@ -0,0 +1,42 @@
    + * @group paragraphs
    

    Probably this is just bikeshedding, but should we create a new field_collection test group for these things?

  12. +++ b/tests/src/Traits/FieldCollectionSourceData.php
    @@ -0,0 +1,134 @@
    +      'paragraphs_item' => [
    ...
    +      'paragraphs_item_revision' => [
    

    Can we add a quick comment here why we have paragraphs data in a field collection data provider?

mikelutz’s picture

Status: Needs work » Needs review
StatusFileSize
new27.84 KB
new8.39 KB

This addresses 1,2,3,4,6,7,9,10, and 12

For #5 and #8, the ids are correct. The *item sources migrate the current revision of each entity, and the entity id uniquely identifies the source row. The *item_revision sources migrate the remaining revisions, and the revision id uniquely identifies the source row. This is important because we are potentially migrating both field collection and paragraph entities into paragraph entities, so all the ids and revision ids are changing and we definitely need the migration maps to map them back up again.The source ids are really only used for mapping, and we definitely want to be able to look up the new entity id without knowing the revision and vice versa. The *item migrations let us map the old entity id to the new entity id without knowing the revision, which is important, because later the revision migration needs to be able to look up the entity by the id and create a new revision off of it.

for #11, I think field_collection has it's own field_collection test @group and we should leave them to it. Our namespace is paragraphs

Status: Needs review » Needs work
mikelutz’s picture

Huh. I guess I need to change the id of the BC class, the test passed locally, so I assumed the extra tags would let the plugin manager select the correct class. I never added a BC class in the first place because I kept the plugin_id and the class should have only been loaded by the plugin id, so I figured the plugin manager would just find the new one no problem. Anyway, try this.

heddn’s picture

No need for an id on the BC class. Its only there if someone extended it. The id and the plugin system will find the non-BC class in its new name.

mikelutz’s picture

You are right, I got confused. This should be deprecated properly.

mikelutz’s picture

Status: Needs work » Needs review
heddn’s picture

Status: Needs review » Needs work

I just discovered ConfigurablePluginInterface yesterday. I suggest we use it for all our configs here in these various source plugins. And I also don't see any doxygen added to sources that have config options. Let's add those docs. Not everyone is going to use an automatic deriver, or even know how to find it. So adding the docs on the classes themselves will only help others and our future selves.

Otherwise, this is looking really close.

mikelutz’s picture

mikelutz’s picture

Status: Needs work » Needs review

Take a look and tell me if that works for you. I added comments and some base classes to implement ConfigurablePluginInterface.

heddn’s picture

Status: Needs review » Needs work

Very close now.

  1. +++ b/src/Plugin/migrate/source/DrupalSqlBase.php
    @@ -0,0 +1,58 @@
    +    // Configuration keys should be snake_case, not CamelCase. Adding a BC
    +    // layer for addDescription which should be depricated.
    

    There's nothing saying config should be snake vs camel. Commerce 2.x uses camel. Its actually not too uncommon in contrib.

  2. +++ b/src/Plugin/migrate/source/d7/FieldCollectionItem.php
    @@ -40,7 +55,7 @@ class FieldCollectionItem extends FieldableEntity {
    +    if (!empty($this->configuration['field_name'])) {
    
    +++ b/src/Plugin/migrate/source/d7/FieldCollectionType.php
    @@ -43,7 +47,7 @@ class FieldCollectionType extends DrupalSqlBase {
    -    if (!empty($this->configuration['addDescription'])) {
    
    +++ b/src/Plugin/migrate/source/d7/ParagraphsItem.php
    @@ -36,7 +51,7 @@ class ParagraphsItem extends FieldableEntity {
    +    if (!empty($this->configuration['bundle'])) {
    

    This can all be simplified to simple checks as I see you did in other places.

    Like: if ($this->configuration['foo'])

mikelutz’s picture

Keeping the new add_description key, but I moved the BC layer and default configurations out of the base plugin and into the two source plugins, and added proper deprecation documentation and error triggering. I simplified the two config checks I missed. (The middle one you posted was a - row : already fixed)

heddn’s picture

Status: Needs work » Reviewed & tested by the community

All looking good. Only very, very small nits. Which could be resolved at commit or left in the code. Commiter discretion.

  1. +++ b/src/Plugin/migrate/source/d7/FieldCollectionItemRevision.php
    @@ -0,0 +1,38 @@
    + *   of that particular type.
    
    +++ b/src/Plugin/migrate/source/d7/FieldCollectionType.php
    @@ -68,8 +89,30 @@ class FieldCollectionType extends DrupalSqlBase {
    +    // layer for addDescription which should be depricated.
    
    +++ b/src/Plugin/migrate/source/d7/ParagraphsType.php
    @@ -0,0 +1,102 @@
    +    // layer for addDescription which should be depricated.
    

    Nit: spelling. Can be fixed on commit.

  2. +++ b/src/Plugin/migrate/source/d7/FieldCollectionType.php
    @@ -68,8 +89,30 @@ class FieldCollectionType extends DrupalSqlBase {
    +    if (isset($configuration['addDescription']) && !isset($configuration['add_description'])) {
    
    +++ b/src/Plugin/migrate/source/d7/ParagraphsType.php
    @@ -0,0 +1,102 @@
    +    if (isset($configuration['addDescription']) && !isset($configuration['add_description'])) {
    

    We should trigger only if addDescription is set. No need to check if add_description too. But no big deal either if left in the code. Could be removed at time of commit.

heddn’s picture

Status: Reviewed & tested by the community » Needs work

There's now enough nits I'm going to unfortunately push back to NW.

+++ b/src/Plugin/migrate/source/d7/FieldCollectionItem.php
@@ -0,0 +1,114 @@
+    $default_configuration = [
...
+    ];
...
+    return NestedArray::mergeDeep(parent::defaultConfiguration(), $default_configuration);

This, and elsewhere, doesn't match the typical method for doing this. This was bugging me when I looked at the code earlier, but I couldn't put my finger on what it was. Below is much more typical.

return [
      'field_name' => '',
    ] + parent::defaultSettings();
mikelutz’s picture

Status: Needs work » Needs review
StatusFileSize
new34.9 KB
new5.26 KB

Alright, I confess to misspelling deprecated a lot, but of, that, particular, and type are all right...

Configurations aren't complex enough to require a nested merge, so I'm fine with simple addition.

My goal was to not override add_description if they were both set, but we definitely need to trigger the error if addDescription is set either way, so I've configured that.

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Phenomenal work here. All feedback addressed.

miro_dietiker’s picture

Status: Reviewed & tested by the community » Fixed

Awesome progress! Committed so we can easily move forward. :-)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.