Hello,

In the past i've been worked with migrate and SQL source format, i can do some import script based on multiple source ID's, and the combination of each make a row unique. Everything work well.

But unfortunately, with XML source, i'm not able to do that, i not dig into the code longly, but i think the problem is that you can specify an unique $item_ID_xpath; in the MigrateSourceXML.

This an exemple of the way i need to use MigrateSourceXML.

// The source ID here is the one retrieved from the XML listing file, and
    // used to identify the specific item's file
    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'ref' => array(
          'type' => 'varchar',
          'length' => 255,
          'not null' => TRUE,
        ),
        'language_code' => array(
          'type' => 'varchar',
          'length' => 255,
          'not null' => TRUE,
        ),
      ),
      MigrateDestinationNode::getKeySchema()
    );


    $item_xpath = '/jobs/job';  // relative to document
    $item_ID_xpath = 'ref';     // relative to item_xpath

    $this->source = new MigrateSourceXML($items_url, $item_xpath, $item_ID_xpath, $fields);

This throw an error on the import, The import is good by the way with only the first ID populated in the migrate table.

I did that change in source.inc, line 256:

          $this->currentKey[$field_name] = $row->$field_name;
        

by

if (isset($row->$field_name)) {
          $this->currentKey[$field_name] = $row->$field_name;
        } else if (isset($row->xml)) {
          $this->currentKey[$field_name] = $row->xml->$field_name->__toString();
        }

And like that its work very well, the two source id are populated and i have no more error after the import, when i update content with an xml import all it's good too.

but i think the problem isn't here, it's more in the way that MigrateSourceXML is implement the sourceID.

Comments

delta’s picture

Title: Can't use mulitple sourceID's with an XML source data » Can't use multiple sourceID's with an XML source data
Status: Active » Closed (fixed)

Seems to work with the 2.5 rc1.. so close this issue. i think it's related to http://drupal.org/node/1701764

delta’s picture

Issue summary: View changes

update description

delta’s picture

I re-open this issue because i think there is a misunderstanding on the use of Migration::map.

Can we use it with an XML source ?

Apparently this seems to be functionnal only with one field. So the class handler to create a map "MigrateSQLMap" doesn't have to be duplicated for XML source and so on accept only one field..?

to correct my issue and use a map with multiple source_id i have to override the migration method : prepareKey().

here my first source_id declared in my_migration->map is 'ref' :

  public function prepareKey($source_key, $row) {
    $key = array();
    foreach ($source_key as $field_name => $field_schema) {
      if($field_name != 'ref') {
        $key[$field_name] = $row->xml->$field_name->__toString();
        continue;
      }
      $key[$field_name] = $row->$field_name;
    }
    return $key;
  }
delta’s picture

Component: Code » Documentation
Status: Closed (fixed) » Needs work
mikeryan’s picture

Category: bug » support
Status: Needs work » Postponed (maintainer needs more info)

Can you describe a little more clearly what you think is wrong? Given that $item_ID_xpath only supports a single value at once, overriding prepareKey is precisely the right thing to do to populate a multi-column key - if that is working for you, then all is as it should be.

mikeryan’s picture

Status: Postponed (maintainer needs more info) » Closed (works as designed)

No further response.

delta’s picture

You are right, thanks for your support !

burningdog’s picture

Status: Closed (works as designed) » Active

I think my problem is similar to the one posted here - at least, my issue title matches.

My source rows key is a combination of 3 fields: Date, StartTime and BroadcastID. Here is an example of my source xml data:

<event>
  <Date>2013/08/31</Date>
  <StartTime>16:40</StartTime>
  <Duration>00:50:00</Duration>
  <BroadcastID>0238435</BroadcastID>
  <ProgrammeTitle>The Young And The Restless</ProgrammeTitle>
</event>

My problem is that although there are over 900 events in the xml, Migrate is only picking up 27 of them. This is because I've set $item_ID_xpath = 'Date'; when creating a new MigrateItemsXML object, and there are 27 unique days in the xml.

I've tried using prepareKey() as suggested ('sourceid' is set up when creating the MigrateSQLMap object):

  public function prepareKey($source_key, $row) {
    $key['sourceid'] = $row->xml->Date . ' ' . $row->xml->StartTime . ' ' . $row->xml->BroadcastID;
    return $key;
  }

I expected that using prepareKey() to set a unique key per row would allow Migrate to pick up all events, but it's still only picking up one event per Date. Am I right to pick prepareKey() for this, or should I be looking somewhere else? I previously had this logic working fine with a csv source file - my source has recently changed to xml.

burningdog’s picture

The solution was to override the way that Migrate accesses a unique row from the XML. Default behaviour is to access it using a single field.

I overrode MigrateItemsXML like so:

class MigrateItemsScheduleXML extends MigrateItemsXML {
  public function __construct($xml_url, $item_xpath='item', $itemID_xpath='id') {
    parent::__construct($xml_url, $item_xpath, $itemID_xpath);
  }

  /**
   * Get the item ID from the itemXML based on itemIDXpath.
   *
   * @return string
   */
  protected function getItemID($itemXML) {
    $itemID = $this->getElementValue($itemXML, 'Date');
    $itemID .= ' ' . $this->getElementValue($itemXML, 'StartTime');
    $itemID .= ' ' . $this->getElementValue($itemXML, 'BroadcastID');
    return $itemID;
  }
}

and then implemented MigrateItemsScheduleXML like so:

    $item_xpath = 'event';  // relative to document
    $item_ID_xpath = 'Date';         // relative to item_xpath and gets assembled
                                         // into full path /producers/producer/sourceid

    $items_class = new MigrateItemsScheduleXML($items_url, $item_xpath, $item_ID_xpath);
    $this->source = new MigrateSourceMultiItems($items_class, $fields);

Thanks to mikeryan's comment at #2 on #1376520: XML with 'concatenated' primary key for pointing me in the right direction.

What's weird is that I can completely discard prepareKey() since that logic is now handled in getItemID() - I'm not sure that this is consistent, since I used prepareKey() to set up my source key when using a csv file, but now I've got to override MigrateItemsXML and getItemID() to do the same logic when using an xml file. That's not obvious.

burningdog’s picture

Issue summary: View changes

Fix typo

mikeryan’s picture

Status: Active » Closed (works as designed)

No open question here.