Migrate field handler (patch)

Thanks becw! How does the patch in #2 account for the non-primary columns in the subfields? For instance, if in the code example, field_label is a filtered text field, how do we handle migrating field_my_multifield:field_label_format? Should those sub-columns be exposed as well?

I guess I'm confused if they would actually be exposed in the UI since the fields() method doesn't drill down to that level. It might benefit to provide some examples of the non-primary columns in the sample code as well.

When testing this module I came up with the following error:
PHP Fatal error: Call to undefined method MultifieldMigrateFieldHandler::import()

Let me describe the test scenario.
- created a multifield Property
- created my migration:
$this->destination = new MultifieldMigrateFieldHandler('property');
- set field mappings

 $this->addFieldMapping('property:field_feature_spec_label', 'propertyLabel');
 $this->addFieldMapping('property:field_specification_value', 'value');
 $this->addFieldMapping('property:field_property_value_unit', 'propertyValueUnit');

Executing drush: drush mi Properties --limit='1 item' gave the error above.

Not sure if this belongs here.

W

@wadem: I don't believe you'd want to use this handler as a destination. You should be using a Node destination or some other kind of entity.

Awesomesauce. Im so glad somebody already did this as its exactly what I need right now.

I applied the patch and tried to run some migrations, execept I was running into some issues with the subfield names not being set correctly. The ->fields method was not handling the subfields properly, so some other calls that migrate was performing was not getting that right.

I modifed the fields method to leverage what MigrateFieldsEntityHandler was already doing for subfields, and now everything is working swimmingly. Im able to migrate data directly into my subfields no problem, *and* Im using the migrate ui (in migrate 2.6).

Im not clear on what all the comment documenation on top of the prepare method is for.. is that explaining what the code is doing? or saying something that I need to do to leverage the code. If its just 'heres what happens below' then we probably dont need that comprehensive of docs. Comments and documentation should explain Why, not What.

I've left them in there though as ultimately thats up to the maintainer if they are useful.

Whoops. Misnamed the patch, also forgot to set the status. These are identical to the comment above.

While I haven't done a proper review of the code, I have tested it and can confirm it is working. Much, much easier than migrating Field Collections. Huge thanks to all involved.

Working on this a bit further. Moved hook_migrate_api() into multifield.migrate.inc and used more API functions for interacting with multifield field types, and converting field items to pseudo-entities and back again.

So I'm experiencing some issues with the migration hander:

1. I'm confused since we seem to still list the multifield itself in the list of destination fields, when we have proper mappings for all of the individual sub-fields. Should we be hiding this? Can you actually migrate values into this destination field?
   
2. Using defaultValue() doesn't seem to work. For example, I have the following in my migration:
   

       $this->addFieldMapping('field_credit_multifield:field_episode_credit_person', 'field_cast')
         ->sourceMigration('ContentPerson'); // This works as expected as it comes from the source data.
       $this->addFieldMapping('field_credit_multifield:field_episode_credit_role') // This is a taxonomy field, and term 271 is valid.
         ->defaultValue(271);
   

   When I run the migration my field value ends up like:

   field_episode_credit_person_target_id (String, 3 characters ): 151
   field_episode_credit_role_tid (NULL)
   

Revised patch that uses multifield_field_info() to register itself for field types so that it will be compatible with #2234769: Should multifield types be represented as separate field types, or just one 'Multifield' field type?

Revised version that properly passes the bundle into MigrateFieldsEntityHandler->fields().

I'm trying to use this to migrate into a multifield using the patch from #22 and becw's example, but I'm getting nothing in my destination. My source D6 field is field_image and my destination is field_image within the field_images multifield (confusing, I know). Here's the code I'm using in my constructor:

    $this->addFieldMapping('field_images:field_image', 'field_image')
      ->sourceMigration('CNSFile');
    $this->addFieldMapping('field_images:field_image:file_class')
      ->defaultValue('MigrateFileFid');
    $this->addFieldMapping('field_images:field_image:preserve_files')
      ->defaultValue(TRUE);

I've verified that the associated images have been migrated in the CNSFile migration and records are there in the migrate_map_cnsfile and file_managed tables. The mappings also show appropriately on the Migrate pages.

After doing some debugging, it looks like the problem is that the default file class - MigrateFileFid in this case - is not available in MigrateFileFieldBaseHandler::prepare():

   if ($migration->getOption('file_function')) {
      $file_class = $migration->getOption('file_function');
    }
    elseif (!empty($arguments['file_class'])) {
      $file_class = $arguments['file_class'];
    }
    else {
      $file_class = 'MigrateFileUri';
    }

even though it is specified in my constructor as the default value for the field_images:field_image:file_class subfield. Is there something else I'm missing?

Thanks.

After some more painful debugging, I think I've found the issue, and it's in Migration::applyMappings():

        // Are we dealing with the primary value of the destination field, or a
        // subfield?
        $destination = explode(':', $destination);
        $destination_field = $destination[0];
        if (isset($destination[1])) {
          $subfield = $destination[1];
          // We're processing the subfield before the primary value, initialize it
          if (!property_exists($this->destinationValues, $destination_field)) {
            $this->destinationValues->$destination_field = array();
          }
          // We have a value, and need to convert to an array so we can add
          // arguments.
          elseif (!is_array($this->destinationValues->$destination_field)) {
            $this->destinationValues->$destination_field = array($this->destinationValues->$destination_field);
          }
          // Add the subfield value to the arguments array.
          $this->destinationValues->{$destination_field}['arguments'][$subfield] = $destination_values;
        }
        // Just the primary value, the first time through for this field, simply
        // set it.
        elseif (!property_exists($this->destinationValues, $destination_field)) {
          $this->destinationValues->$destination_field = $destination_values;
        }
        // We've seen a subfield, so add as an array value.
        else {
          $this->destinationValues->{$destination_field} = array_merge(
            (array)$destination_values, $this->destinationValues->{$destination_field});
        }

Basically, this code is only designed to handle one level of subfields (there are only specific references to $destination[0] and $destination[1], not a loop through $destination), so what's happening is that the defaultValue value is overriding the fid value (in this case) at this line:

  // Add the subfield value to the arguments array.
  $this->destinationValues->{$destination_field}['arguments'][$subfield] = $destination_values;

So the first time through, $this->destinationValues->field_images['arguments']['field_image'], has the fid value I want, but after going through the file_class subfield, it is replaced by 'MigrateFileFid'.

And then, it goes through the preserve_files subfield (true), and resets to true. This is the "true" value that I am seeing when the code gets to the handler, so this explains everything perfectly.

So with that being the case, is there a way to handle a subfield for a field within a multifield, or will it require changes in migrate itself? Since the value is already set by the time it gets to the handler, I'm guessing it's a change to migrate.

Thanks

Attached is a patch that updates the handler to work in conjunction with a patch to Migrate module that I wrote that handles three levels of subfields. As mentioned above, however, this patch is irrelevant without the Migrate patch, since currently Migration::applyMappings() stomps on the third level subfields, so there's nothing to process when the migration process gets to this handler.

I have a migration to an image field within a multifield working perfectly with these two patches.

After multiple hours of debugging and trying to figure out why null values were being passed to file_usage_add() for an image field in my multifield when there were no files in the source, I determined that the cause was that this handler was returning NULL instead of an empty $return array. Simply adding

$return = array();

fixed the problem. An updated patch is attached.

Sigh.

If migrating a multivalued multifield, the $subfield_arguments variable gets overridden on every iteration of the loop.
Fixed this by adding the $key variable and turning it into an array.

If migrating a multivalued multifield, the $subfield_arguments variable gets overridden on every iteration of the loop.
Fixed this by adding the $key variable and turning it into an array.

I have missed commenting out a line in the previous patch.
If this is set then only the first value of the multivalued field will get imported. So needs removing.
$arguments[$key] = $arguments[$key][0];

I'm running a migration, so I'll RTBC the patch ASAP.

Patch #30 works great with Migrate 2.8-beta1.

I've added support for entity translation to the multifield handler. I've tested it on my part and it works.

looks and works great!

+++ b/multifield.migrate.inc
@@ -0,0 +1,190 @@
+          $values['arguments'] = $subfield_arguments[$subfield_name];

There are instances where this is not defined. The assignment should be protected with a if statement and an array_key_exists. I'll resubmit shortly

I'm putting it back into RTBC because the changes made are relatively trivial, but feel free to review it again.

The documentation for how to make this work could use some help. I'm dealing with a unlimited number of multifield entries (each entry still is limited to a single value, per the limitation for multifield). In my prepareRow, what do I need to create and how does that look? The documentation in #13 / #15 isn't helping.

This improves the documentation. Since it is a documentation only change, marking RTBC again since it was already RTBC previously.

Thanks for the work on this, patch from #39 works great for me.

I have had problems migrating titles of the link filed. The problem is, that all subfield arguments are passed to the field handler (not sure if I am using the correct terminology), as the delta of the link does not match the delta of the arguments (because delta is always 0).

Attached patch fixed this the way, that only the arguments corresponding to the delta are passed. I have only tested with link title, not with other subfields, and only a few cases. Does the code make sense? Do we need to check first, if $subfield_arguments[$subfield_name] is an array, or will this alway be an array?

Checked and ran well on a migration we have.

I've got migration 7.x-2.8 installed and the the latest patch (41) on this thread, and now I'm trying to migrate multivalue multifield consiting of a taxonomy reference + a text field.

Here is the relevant code in my migration module:

    $this->addFieldMapping('field_user_mm_technical_skills:field_user_mm_ts_skill','technical_skill')
      ->separator(',')
      ->sourceMigration('Technical','Technical2');
    $this->addFieldMapping('field_user_mm_technical_skills:field_user_mm_ts_remarks','technical_remarks')
      ->separator('|/|/|');
    $this->addFieldMapping('field_user_mm_technical_skills:field_user_mm_ts_remarks:language')
      ->defaultValue('en');
    $this->addFieldMapping('field_user_mm_technical_skills:field_user_mm_ts_remarks:format')
      ->defaultValue('plain_text');
    $this->addFieldMapping('field_user_mm_technical_skills:field_user_mm_ts_skill:source_type')
      ->defaultValue('tid');

The user in my test case should have 8 entries in this multifield. When I run the migration I get the following in the message log:

395	Informational	Uninitialized string offset: 2 File /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156(file: /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156)
395	Informational	Uninitialized string offset: 3 File /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156(file: /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156)
395	Informational	Uninitialized string offset: 4 File /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156(file: /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156)
395	Informational	Uninitialized string offset: 5 File /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156(file: /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156)
395	Informational	Uninitialized string offset: 6 File /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156(file: /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156)
395	Informational	Uninitialized string offset: 7 File /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156(file: /Users/bob/ultrabob.prv/www/sites/all/modules/multifield/multifield.migrate.inc, line 156)

$row->technical_skills has the value 105,109,110,135,147,146,145,139
and $row->technical_remarks has the value |/|/||/|/||/|/||/|/||/|/||/|/||/|/| meaning that this user didn't add any text details.

Nothing is migrated into the multifield. Am I coding this wrong, or does the handler not know how to handle multiple entries passed in with the separator method, or did I apply the wrong patch? Any help greatly appreciated! I'm pretty new to drupal and the community, so if I'm going about this wrong, please educate me.

You might have to run the code in a debugger and step through it to see how migrate deals with a bunch of empty values. You also might have to add a callback to your mapping in the meantime.

Just as a quick followup, I did some more to try to narrow things down, by commenting out first the code for the text field part of the migration, and then the taxonomy reference part on a different user with remarks in place. When I ran the migration with just the taxonomy reference there was nothing in the messages log, and also nothing in the technical skills multifield. When I ran the migration with just the text field, the message log was populated again and now there were 6 blank fields(8 entries were in the source list).

It seems like I have two seperate issues, both including use of separator. One with the text fields, blank values are coming through. Second, with the use of sourceMigration() nothing comes through. If I run the taxonomy ref migration without sourceMigration() in place one of the two test users gets 2 skills filled out of 8, though only one of them is exactly right, the other is a parent of the selected term. (I suspect I just got lucky with some ids in common, but I don't know)

I don't feel confident enough that my change will not break things for others, because I think the line must be written the way it is to address other cases, perhaps where a fieldMapping has more than just one argument. Also being new to drupal I'm not really sure or the details of how to submit a patch. Still, here is what I changed to make the multifield work for me in my situation with one translatable text field, and one taxonomy reference, coming from a sourceMigration with sourceType tid.

Line 156:

$values['arguments'][$subfield_argument_key][] = $subfield_argument[$delta];

was assigning a value of t, then i, then d and it was assigning the value one level deeper than it was needed. I changed that line to:

$values['arguments'][$subfield_argument_key] = $subfield_argument;

and now it works for me, with my set of fields. Could someone look at that and see if it applies more generally, or whether there is a need for a conditional there that checks for whether the value is an array or not?

Here is a replacement for that line that will not change behavior in cases where $subfield_argument is an array as seems to have previously been expected, but will fix the case where it is not an array. If that behavior was not a bug in some cases, this will be better.

              if (is_array($subfield_argument)) {
                $values['arguments'][$subfield_argument_key][] = $subfield_argument[$delta];
              } else {
                $values['arguments'][$subfield_argument_key] = $subfield_argument;
              }

Thanks for this awesome patch! I'm so glad that I didn't have to roll something of my own. However I did have one small problem with it. I have a multifield that among other things contains an integer field. Let's call it score Whenever migrate was a creating or updating a node without a score, I was getting SQL errors stating that an invalid value was being set for scores column in the database.

What I did to work around this was set the scores to an impossible value (-99) in the migrations prepareRow() method if they were null in the source. Then I added a prepare() method that does the following

  public function prepare($entity, stdClass $row) {
    for ($delta = 0; $delta < 2; $delta++) {
      if ($entity->field_teams[LANGUAGE_NONE][$delta]['field_score'][LANGUAGE_NONE][0]['value'] == -99) {
        $entity->field_teams[LANGUAGE_NONE][$delta]['field_score'][LANGUAGE_NONE] = array();
      }
    }
  }

I suspect that what was originally happening is that $entity->field_teams[LANGUAGE_NONE][$delta]['field_score'][LANGUAGE_NONE] was equal to

    array(0 => array('value' => NULL))

instead of

    array()

Thanks @becw, that is just what I was addressing in #47. I wish I could test this for you, but there has been so little happening with multifield, that I abandoned it, and wrestled with field collection until it worked for me. I can verify that the last part of that patch works, because you fixed it exactly the same way I did.

I'm running a migrate against this with multiple values going to it. I am migrating in a taxonomy term to value pairing. The taxonomy term was already migrated previously. And each node has 4 or 5 of these key/value pairings. So the pairings consist of the term as the key and a text field as the value.

I am pumping in an array of these values to two mapped fields and have mapped to the correct locations. However, I am seeing that the migrate script is not getting the data correctly. We have the tid for the term going to the fields.inc in the migrate module. But it does not recognize it because the $arguments['source_type'] returns an array instead of a flat value.

Further, when I use xdebug to examine during a migration I am noticing that the $return variable in this plugin comes back structured in such a way that it does not have a value in the field_attribute for each. Just an empty array.

Example of the $return variable structure (see attached screenshot).

I believe something is wrong with how this is being structured. I am still debugging to figure out how to fix it. I am using the most recent patch from comment #50.

Ok guys, the problem had to do with how the data is being structures in the $values array. When working with a source_type of 'tid' it expects the source type to be a flat value, not an array.

So to fix it I simply checked for that situation and it solves the problem.

I have attached the interdiff and the new patch that is working for me. Let me know if you have any questions. Also I did some visual cleanup to make the code more readable.

So apparently the first patch I made didn't work so well. Here is attempt number two and the same/previous interdiff should still apply.

Thank you for this!