D6 Taxonomy Term Reference Field is not migrated to D8 Field [#2884240]

Update

The described need for a process pipeline other then GET is only necessary for Drupal 6 "Content Taxonomy" fields (contrib module). If the D6 Vocabulary is attached to a content type by Drupal 6 core means only, the process pipeline can be written in the simplest form of a GET plugin. "drush migrate-upgrade --configure-only" creates all the necessary configuration for creating the D8 term reference field and migrating all values during the subsequent migration. More details to this workaround here: #2884240-4: D6 Taxonomy Term Reference Field is not migrated to D8 Field
Even most uses of a Content Taxonomy field can be made compatible for core migration by adding the field settings to the D6 vocabulary administration page (i.e. check content types and check single/multiple values).
Nonetheless, a direct support of Content Taxonomy by D8 migration will simplify migration in these cases. There is a feature request issue for adding support for Content Taxonomy here: #2885298: Support content_taxonomy term reference fields from D6.

Reproducing the bug

1. Under Drupal 6 (incl. CCK) create a content type with a taxonomy term reference field using a vocabulary and some records with terms selected.
2. Under Drupal 8 create the vocabulary with the same terms (manually) and a custom module migrate_d6 - create an empty new module using drupal console - with a migration yml file consisting of at least (I'm omitting all other standard fields of a D6 node migration for simplicity, also omitting the migration group yml file):

modules/custom/migrate_d6/config/install/migrate_plus.migration.node_migration.yml

id: node_migration
label: Node Migration
migration_tags:
  - Drupal 6
deriver: Drupal\node\Plugin\migrate\D6NodeDeriver
migration_group: migration
source:
  plugin: d6_node
process:
  field_terms: field_terms
destination:
  plugin: entity:node

3. Make sure, you have the core migrate modules enabled and the migrate_plus and migrate_tools contributed modules downloaded and enabled.
4. Enable your custom migrate_d6 module.
5. Migrate the D6 nodes to D8 via

$ drush migrate-import node_migration

Expected behaviour

I would expect the D6 nodes to be migrated, including all taxonomy terms referenced in the field_terms field.

Buggy behaviour

To my surprise the taxonomy term reference fields were completely empty in D8, despite the other D6 standard fields being properly migrated to D8.

Debugging status info

After some days of debugging and starting to learn first basics of migration plugins, I found that the d6 source plugin delivers the taxonomy reference terms in the following nested array to the process plugin (example):

            [field_terms] => Array
                (
                    [1] => Array
                        (
                            [delta] => 1
                            [value] => 4
                        )

                    [0] => Array
                        (
                            [delta] => 0
                            [value] => 2
                        )

                )

This case is an example of two taxonomy terms with the tids 4 and 2.

I found out, that the D8 destination plugin expects this in a different array structure by testing with a default_value-process plugin (example):

            [field_terms] => Array
                (
                    [0] => 4
                    [1] => 2
                )

I don't know why the D8 destination plugin doesn't accept the nested array delivered by the D6 source plugin, I assume this is a bug.

Workaround for the time being

After several days of debugging I found a workaround by using two process plugins instead of the implicit get-process plugin used in the above yml file:

modules/custom/migrate_d6/config/install/migrate_plus.migration.node_migration.yml

id: node_migration
label: Node Migration
migration_tags:
  - Drupal 6
deriver: Drupal\node\Plugin\migrate\D6NodeDeriver
migration_group: migration
source:
  plugin: d6_node
process:
  field_terms:
    -
      plugin: iterator
      source: field_terms
      process:
        id: value
    -
      plugin: flatten
destination:
  plugin: entity:node

What am I doing in the process field_terms-destination?

I use the iterator-process plugin to loop through every taxonomy term sub-array in the source array and keep only the 'value' key value and assign it to a temporary 'id' key. 'id' could be any other name, the exact name is not important.
Then I use the flatten-process plugin to move the values from the sub-array into the first level of the array, practically removing the sub-arrays including the temporary 'id' keys.

By this the migration of the D6 taxonomy term references to D8 seems to work just fine. Please be aware that the term tids are not changed in this example between D6 and D8 as this would require a migration_lookup-process plugin to be used.

But, shouldn't there be a default behaviour, that the D6 source plugin delivers an array format the D8 destination plugin can save? Or am I misunderstanding the migration of taxonomy term references. The answers to these two questions decide whether this is really a bug report or a support request.

Thank you very much!

Comments

Comment #1

7 June 2017 at 00:39

meichr created an issue. See original summary.

Comment #2

mikeryan

he/him

English

Pittsfield, MA, USA

commented 7 June 2017 at 15:04

Category:	Bug report	» Support request
Status:	Active	» Postponed (maintainer needs more info)

The sub_process (née iterator) process plugin is indeed what you need to use here (although flatten puzzles me - did you try it without flatten?). If you do a drush migrate-upgrade --configure-only to see how the deriver creates the field mappings, you'll see that CCK field mappings are generated using iterator.

My usual approach to a D6 or D7 migration to D8, to make sure I'm picking up the nuances of CCK field mappings (which can be a bit convoluted for the more complex fields), is to do that --configure-only to generate starting points for my migrations, then modify them for the particular use case.

Comment #3

meichr commented 8 June 2017 at 23:28

Version:	8.3.2	» 8.3.3
Status:	Postponed (maintainer needs more info)	» Active

Hi mikeryan, thanks a lot for these tips and the --configure-only option, previously I had copied templates from the node and taxonomy modules. In the following are my findings:

1.) drush mi --configure-only
I installed a fresh D6.38 website, added the cck and taxonomy_content modules and a field_term field to the Page content type assigned to a new vocabulary "Terms". Then I created some test terms and test content (Pages) and assigned a number of terms to each page.
I installed a fresh D8.3.3 website and downloaded and enabled modules migrate_upgrade and migrate_tools using drush. I added a database[upgrade][default] configuration to the settings.php.
I used "drush mi --configure-only" (thank you) to create the migration configuration.
Then I used "drush cex --destination= to export all the configuration including "migrate_plus.migration.upgrade_d6_node_page.yml".

Result
field_terms had been added to the file "migrate_plus.migration.upgrade_d6_node_page.yml", but only as "field_terms: field_terms". Trying to migrate the pages to Drupal 8.3.3 using this migration yml in a custom module failed (even though the vocabulary and terms were available, either migrated or manually created). It seems the migrate_upgrade module doesn't recognize the D6 Content Taxonomy fields properly, or, at least doesn't know that the iterator plugin is needed.

2.) Removing the flatten plugin from my manual migration configuration in the issue summary

Instead of

process:
  field_terms:
    -
      plugin: iterator
      source: field_terms
      process:
        id: value
    -
      plugin: flatten

I tested

process:
  field_terms:
    -
      plugin: iterator
      source: field_terms
      process:
        id: value

Result:

The source array of D6 taxonomy terms referenced by field_terms in the page content type had been (same example as in the issue summary):

            [field_terms] => Array
                (
                    [1] => Array
                        (
                            [delta] => 1
                            [value] => 4
                        )

                    [0] => Array
                        (
                            [delta] => 0
                            [value] => 2
                        )

                )

After processing the array looked like

            [field_terms] => Array
                (
                    [0] => Array
                        (
                            [id] => 4
                        )

                    [1] => Array
                        (
                            [id] => 2
                        )

                )

But the destination plugin did not save this array to the D8 page content, the Terms field stays empty for every page. Also, there is no message (neither by "drush migrate-messages" nor as output from the "drush mi" command) that there would be anything wrong with the field_terms migration.

When I leave the flatten plugin in the migration configuration as in the issue summary, then after processing the array looks like

            [field_terms] => Array
                (
                    [0] => 4
                    [1] => 2
                )

And only a field_terms array in this format is properly saved by the destination plugin.

I also did have a second proof that this latter array is the correct output of the process plugins, if I test the same with the default_value process plugin:

process:
  field_terms:
    -
      plugin: default_value
        - 4
        - 2

If I to a "drush mi" with this process plugin I receive the following array:

            [field_terms] => Array
                (
                    [0] => 4
                    [1] => 2
                )

And also here, this array is properly saved to the D8 page field_terms.

Resumé
1.) Wouldn't it be valuable, if "drush mi --configure-only" would create already the proper process plugin pipeline instead of creating configuration for the non-working get process plugin for D6 taxonomy term fields?
2.) Is the proper plugin pipeline now correctly iteration and flatten or is there a third version which really is correct, like using a certain key instead of "id" which I used, or else?

Thank you again for your valuable help, I learn a lot out of experimenting with this, though, there really is a big D6 site to be migrated in the end ;-). I leave this as a support request, though it could be a feature request or again a bug report in case of Resumé 1.) .

Comment #4

meichr commented 19 June 2017 at 01:51

Workaround

Content_Taxonomy-fields in Drupal 6 are fields to store term references as a field instead of attaching a vocabulary to one or more content types.

If in your Drupal 6 website there is only one Content_Taxonomy-field per vocabulary attached to a content type, then the following workaround can be used for migration of a Content_Taxonomy field:

Create a copy of your Drupal 6 website as source for migration, you may have done this already, anyway.
Administer each vocabulary for which a Content_Taxonomy field is attached to one or more content types.
Check each content type check box in the vocabulary administration page, to which a Content_Taxonomy field has been created.
Make sure, the "Multiple selection" check box is set the same as the Content_Taxonomy field. Also here only one value is allowed, but that should be no problem, as it can be set to multiple for all fields, if at least one is multiple.
Now you can create the migration configuration with the drush command "drush migrate-upgrade --configure-only". You need the Migrate Upgrade and Migrate Tools modules enabled for this.
The migration itself - "drush mi --all" - then creates the term reference fields at each content type using the machine name of the vocabulary as the term reference field machine name. You can't rename the machine name, if there is already data migrated, so this may be an obstacle, if you need the original field machine name.

These are the constraints for the workaround again:

Only one term reference field per vocabulary and content type
The created term reference field in Drupal 8 has a different machine name than in Drupal 6, i.e. the D6 vocabulary machine name
All term reference fields per vocabulary are migrated as either single or multiple field, the second, if at least one term reference field is multiple selectable.

When D6 Content_Taxonomy fields are supported in the future, this additional work becomes unessecary and the constraints are obsolete.