I get the following output while running the migrator through drush (from D6):

Attempt to create a field storage with an name longer than 32 characters: profile_expertise_electoralsystem        [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Attempt to create a field storage with an name longer than 32 characters: profile_expertise_executivesystem        [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Attempt to create a field storage with an name longer than 32 characters: profile_expertise_legislativesystem      [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Attempt to create a field storage with an name longer than 32 characters: profile_expertise_naturalresources       [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Processed 39 items (35 created, 0 updated, 4 failed, 0 ignored) - done with 'upgrade_user_profile_field'           [status]

I didn't make any tweaks to upgrade_user_profile_field[_instance].yml before running.

Shouldn't the configure step/plugin look for this limit and truncate the target field name? Is there a way for me to map the names?

P.S. These are checkboxes defined through the D6 Profiles module.

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

niccottrell created an issue. See original summary.

mikeryan’s picture

Project: Migrate Upgrade » Drupal core
Version: 8.x-2.0-rc1 » 8.2.x-dev
Component: Code » migration system

Shouldn't the configure step/plugin look for this limit and truncate the target field name?

Yes, the core user_profile_field.yml migration plugin should be using the dedupe_entity process plugin here.

Is there a way for me to map the names?

Until that's fixed, in your own upgrade_user_profile_field you can use dedupe_entity to automatically truncate and maintain uniqueness of your field names:

  field_name: 
    plugin: dedupe_entity
    source: name
    length: 32

If you'd like to control the resulting field names, you can use static_map:

  field_name:
    plugin: static_map
    source: name
    bypass: true
    map:
      profile_expertise_electoralsystem: field_electoralsystem
      profile_expertise_executivesystem : field_executivesystem
      profile_expertise_legislativesystem: field_legislativesystem
      profile_expertise_naturalresources: field_naturalresources

Note that either way, you will of course need to change references to these fields in other migrations such as d6_profile_values.

niccottrell’s picture

Thanks @mikeryan. I made these changes and recently tried a fresh migrate from the D6 site using the dedupe_entity plugin. I got a different error this time:

exception 'Drupal\Component\Plugin\Exception\PluginNotFoundException' with message 'The "" entity type does  [error]
not exist.' in /www/cn8/core/lib/Drupal/Core/Entity/EntityTypeManager.php:125
Stack trace:
#0 /www/cn8/core/lib/Drupal/Core/Entity/EntityTypeManager.php(225):
Drupal\Core\Entity\EntityTypeManager->getDefinition(NULL)
#1 /www/cn8/core/lib/Drupal/Core/Entity/EntityTypeManager.php(161):
Drupal\Core\Entity\EntityTypeManager->getHandler(NULL, 'storage')
#2 /www/cn8/core/lib/Drupal/Core/Entity/EntityManager.php(62):
Drupal\Core\Entity\EntityTypeManager->getStorage(NULL)
#3 /www/cn8/core/lib/Drupal/Core/Entity/Query/QueryFactory.php(54):
Drupal\Core\Entity\EntityManager->getStorage(NULL)
#4 /www/cn8/core/modules/migrate/src/Plugin/migrate/process/DedupeEntity.php(56):
Drupal\Core\Entity\Query\QueryFactory->get(NULL, 'AND')
#5 /www/cn8/core/modules/migrate/src/Plugin/migrate/process/DedupeBase.php(40):
Drupal\migrate\Plugin\migrate\process\DedupeEntity->exists('profile_fullnam...')
#6 /www/cn8/core/modules/migrate/src/MigrateExecutable.php(398):
Drupal\migrate\Plugin\migrate\process\DedupeBase->transform('profile_fullnam...',
Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'type')
#7 /www/cn8/core/modules/migrate/src/MigrateExecutable.php(217):
Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row))
#8 [internal function]: Drupal\migrate\MigrateExecutable->import()
#9 phar:///usr/local/bin/drush/includes/drush.inc(720): call_user_func_array(Array, Array)
#10 phar:///usr/local/bin/drush/includes/drush.inc(711): drush_call_user_func_array(Array, Array)
#11 /www/cn8/modules/migrate_tools/migrate_tools.drush.inc(269): drush_op(Array)
#12 [internal function]: _drush_migrate_tools_execute_migration(Object(Drupal\migrate\Plugin\Migration),
'upgrade_user_pr...', Array)
#13 /www/cn8/modules/migrate_tools/migrate_tools.drush.inc(235): array_walk(Array, '_drush_migrate_...',
Array)
#14 [internal function]: drush_migrate_tools_migrate_import()
#15 phar:///usr/local/bin/drush/includes/command.inc(366): call_user_func_array('drush_migrate_t...', Array)
#16 phar:///usr/local/bin/drush/includes/command.inc(217): _drush_invoke_hooks(Array, Array)
#17 [internal function]: drush_command()
#18 phar:///usr/local/bin/drush/includes/command.inc(185): call_user_func_array('drush_command', Array)
#19 phar:///usr/local/bin/drush/lib/Drush/Boot/BaseBoot.php(67): drush_dispatch(Array)
#20 phar:///usr/local/bin/drush/includes/preflight.inc(66): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#21 phar:///usr/local/bin/drush/includes/startup.inc(321): drush_main()
#22 phar:///usr/local/bin/drush/drush(114): drush_startup(Array)
#23 /usr/local/bin/drush(10): require('phar:///usr/loc...')
#24 {main}

I've attached the config for this fullname field, which is not a required field.

User profile fields

Fullname field config

I don't see why a simple textfield field should be a problem. What can I do to help track this down?

mikeryan’s picture

Status: Active » Postponed (maintainer needs more info)

Could you post your current migration configuration (.yml)?

Thanks.

niccottrell’s picture

mikeryan’s picture

Status: Postponed (maintainer needs more info) » Active

Don't forget to set the issue status back to "Active" when providing more info to a "Postponed (maintainer needs more info)".

mikeryan’s picture

The (immediate) problem is here in the field migration:

  field_name: name
  type:
    plugin: dedupe_entity
    source: name
    length: 32
    map:
      checkbox: boolean
      date: datetime
      list: text
      selection: list_string
      textfield: text
      textarea: text_long
      url: link

Please refer back to comment #2 above - the dedupe_entity plugin needs to be applied to field_name, not to type (which needs to be returned to its original configuration using static_map). You'll also need to apply the same transformation to field_name in the field_instance migration.

Now, that being said, I dug in a little farther and while the above should fix the creation of the profile fields, it looks like the migration of the actual values is using the names directly (not referring to the rewriting in the field migrations), so it doesn't look like that will work. Offhand, I don't see a way around that without writing an extension of the d6_profile_field_values source plugin.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

Issue tags: +migrate-d6-d8, +migrate-d7-d8
quietone’s picture

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Olarin’s picture

Priority: Normal » Major

Capital letters in the field machine name will also break the migration. I assume both problems could both be addressed simultaneously? Otherwise that could be spun off into a separate issue of course.

Olarin’s picture

OK, so I'm trying to pick at this myself and I understand at least some of what needs to be done here, but I may need some assistance for the rest:

First of all, user_profile_field.yml and user_profile_field_instance.yml need to be modified, as indicated above by mikeryan, to ensure that the field machine name is a valid drupal 8 machine name, and that there are no duplicates. I know how to do the first part - in the process section, this needs to be added:

field_name:
  -
    plugin: machine_name
    source: name

and then that needs to chain into something to prevent duplicates - I presume that would be make_unique_entity_field which appears to have replaced the confusingly named dedupe_entity that was referenced above - but I'm not quite sure how to configure it (in particular I'm at a loss as to what the "field" parameter to make_unique_entity_field should be).

Then the second half, again as mentioned above, is to use migration_lookup to transform the field names when you're actually bringing in the values, in the d6_profile_values migration (and presumably also in some equivalent somewhere for Drupal 7, but I'm not yet sure what that is). This is the other place I get stuck right now, because d6_profile_values brings all the fields in at once in one row, so either the migration lookup needs to be called from the source plugin (which I'm unclear on how to do or even if it's valid to do), or the whole migration needs to be restructured somehow?

quietone’s picture

I haven't read the whole issue but maybe I can help clarify the two the make_unique_entity_field and migration_lookup process plugins.

The field property in the make_unique_entity_field process plugin. The field is just any field that the entity has. If the entity has properties 'id', 'name' and 'color', any of those can be used as the field you want to make unique. Obviously, context is important and making a color unique doesn't make much sense, but the process plugin would do it. The plugin is normally used to make sure that a key is unique.

In this excerpt from d6_taxonomy_vocabulary.yml, the 'vid' field is made unique, only when it is a migrated entity (existing non migrated vocabularies are not changed and limited to 32 characters. The resulting value is then piped to the forum_vocabulary process plugin.

  vid:
    -
      plugin: machine_name
      source: name
    -
      plugin: make_unique_entity_field
      entity_type: taxonomy_vocabulary
      field: vid
      length: 32
      migrated: true
    -
      # This plugin checks if the vocabulary being migrated is the one used by
      # Forum. If so, we use the machine name that Forum expects. Otherwise, we
      # leave it unchanged.
      plugin: forum_vocabulary
      machine_name: forums

The d6_user migrations has an example of migration_lookup :

  user_picture:
    plugin: migration_lookup
    migration: d6_user_picture_file
    source: uid
    no_stub: true

Here the uid property of the existing row is the input to migration_lookup, which takes that input and determines what the result that a previous run of d6_user_picture_file returned for that same input uid. So, if the previous run d6_user_picture_file had an input of '5' and returned '24', then if the uid was 5 in this example, then 24 would be saved in the 'user_picture' field. This helps ensure that your destination site keeps things in sync, in this case the user picture fid for a user.

Here is a link to the api doc for all the process plugins, which may be useful; https://api.drupal.org/api/drupal/namespace/Drupal%21migrate%21Plugin%21...

Olarin’s picture

OK, well apparently make_unique_entity_field is not the correct plugin, then. In this case, it's not a value *in* a field we want to ensure uniqueness of, it's the name of the field itself, which we're just now creating, that we want to ensure uniqueness of. (See, we might have had profile fields of both FirstName and firstname in Drupal 6, because it allowed machine names with capital letters. Drupal 8 doesn't, so we'll run the field names through the machine_name process plugin first, but now we risk duplicates (in this case two fields both named firstname), so we need to do something to make sure each field still gets a unique name (and also truncate them to 32 characters, which was the original point of this issue).)

As for migration_lookup, I did read the documentation and I do understand how it works. The problem is again that it's not a *value* we need to do a lookup for, it's actually the name of the field itself. (Once user_profile_field and user_profile_field_instance are modifying profile field machine names to be valid d8 machine names, d6_profile_values will need to make sure it's saving those values to the new names of the fields, not the old - possibly different and invalid - names.)

mradcliffe’s picture

I talked with @Olarin about this issue, and we came to the following conclusions:

1. We need to create a new process plugin to guarantee length and uniqueness. MakeUniqueBase does not guarantee length and can output values greater than the provided length.
2. The profile field migration template needs to also have the machine_name plugin process the name to guarantee allowable characters.
3. We need to use the MigrationInterface object from the base class in ProfileFieldValues source plugin because process plugins only transform the value (field value), not the key (field name) and ProfileFieldValues source plugin explicitly sets the key.

mradcliffe’s picture

So after writing that and not writing my first instinct which was to inject migration plugin manager, I need to do that, because profile field isn't profile field values migration.

So maybe a trait makes sense to share code from MigrationLookup?

Olarin’s picture

Status: Active » Needs review
FileSize
12.87 KB

@mradcliffe and I worked on this yesterday, and finally got something working. Here's my first attempt at compiling our work into a patch. This is against 8.4, I'll try to roll against 8.5 and 8.6 momentarily.

To review: we wrote a new process plugin for user_profile_field to use (along with the existing machine_name plugin) to ensure that profile field names are munged into something valid for Drupal 8, had user_profile_field_instance look up and use the mapping that user_profile_field created, and modified the source plugin for d6_profile_values to make sure it also looks up those mappings for the field names when it prepares the row of values to be saved.

I found that I also needed (a modified version of) the fix in https://www.drupal.org/project/migrate_upgrade/issues/2823414#comment-11... in order to actually get the values to import in my test setup. (For the record, I'm using migrate-upgrade --configure-only and then using migrate-import for the specific migrations I want, sometimes one at a time, which according to that issue queue might be part of why I'm still experiencing that bug.) I don't know if setting the destination property in the source plugin is a "correct" solution for that problem; if not I'd welcome suggestions in that regard.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Olarin’s picture

Version: 8.5.x-dev » 8.6.x-dev
FileSize
12.77 KB

Here's an 8.6.x patch, identical except for the migration_templates => migrations folder name change.

Olarin’s picture

Aaaand testing the same patch against 8.5.x.

heddn’s picture

Priority: Major » Normal
Issue tags: +Needs tests

I'm not sure why 'make_unique_entity_field' won't work for this. Can we expound on that?

Based on https://www.drupal.org/core/issue-priority, I'm dropping the priority down to normal since this doesn't seem to effect a wide number of users.

heddn’s picture

Status: Needs review » Needs work

Also needs work because needs tests.

Olarin’s picture

From the documentation for MakeUniqueEntityField:

"Ensures the source value is made unique against an entity field."

Key word here being *value*. It's made to ensure that the values stored in a field are unique, and it in fact requires the name of the field as a configuration key. What we're trying to process in this issue is the *name* of the field (in user_profile_field), not the values we will later put in it (in d6_profile_values).

I'm not sure how else to expound upon it. Unless I'm completely failing to understand the entire situation (in which case I would of course welcome some education), make_unique_entity_field is made for a fundamentally different use case than what we're dealing with here.

(There's also the additional problem that make_unique_entity_field doesn't actually guarantee ouptut length; the length parameter is only used to truncate the input, not to limit the length of the output - the code that ensures uniqueness for make_unique_entity_field can still add an arbitrary number of characters to the output.)