In MigrateItemsXML::getIDsFromXML(…) the XPath expressions itemXpath and itemIDXpath get concatenated like this:

$full_xpath = $this->itemXpath . '/' . $this->itemIDXpath;

This can result in wrong behaviour. Consider this case:

$itemXpath = '/node_export/node[nid = tnid]/field_camp_images/*[not(@type)]' .
    ' | /node_export/node[nid = tnid]/field_camp_gallery_images/*[not(@type)]';
$itemIDXpath = 'fid';
$full_xpath = $itemXpath . '/' . $itemIDXpath;
// results in '/node_export/node[nid = tnid]/field_camp_images/*[not(@type)] | /node_export/node[nid = tnid]/field_camp_gallery_images/*[not(@type)]/fid'

In this case the wrong number of IDs gets reported.

Comments

Pisco’s picture

Status: Active » Needs review
StatusFileSize
new1.72 KB

The solution is to iterate over the elements (itemXpath) and call getItemID(…) to get the id. Funny enough it's done exactly like this in getItemsFromXML(…).

getIDsFromXML(…) and getItemsFromXML(…) now look very similar, maybe we can reuse the code instead of duplicating it?

Pisco’s picture

Note however, that you can work around this bug by using the following expression in my example:

$itemXpath = '/node_export/node[nid = tnid]/*[(name() = "field_camp_images") or (name() = "field_camp_gallery_images")]/*[not(@type)]';
mikeryan’s picture

Status: Needs review » Fixed

Committed, thanks!

Pisco’s picture

Status: Fixed » Needs review
StatusFileSize
new3.25 KB

I have to thank you for an absolutely awesome module! Thank you!

I noticed that you maybe accidentally committed the patch file itself (fix-xpath-handling-1054616-1.patch).

How about code reuse of getItemsFromXML(…) (see my question above)? I think it would be much more robust and maintainable if getIDsFromXML(…) would call getItemsFromXML(…). They do essentially the same. Performance wise I think it wouldn't be much of a tradeoff.

drewish’s picture

You don't need the call to array_unique() any more:

+    $this->cache_ids = array_unique(array_keys($ids));

because keys are unique by definition.

drewish’s picture

Status: Needs review » Needs work

But I'm not sure it makes sense to separate them out... or if you're going to go that way we should drop the getItemID() functions.

Pisco’s picture

StatusFileSize
new3.23 KB

@drewish, I don't think I understand, what do you mean by “separate them out”? And why drop getItemID(), because of the extra function call? I don't think you gain anything by dropping it.

The patch improves maintainability and code reuse, nothing more.

drewish’s picture

Yeah I missing that getItemID() was used by getItemsFromXML(). But I think the current process makes more sense that potentially bundling up a huge number of items just to have the caller discard them and pull the keys back out.

Pisco’s picture

Status: Needs work » Fixed

As you like of course. Thanks for looking into it!

I think the issue can be considered fixed now.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

twod’s picture

Status: Closed (fixed) » Needs review
StatusFileSize
new2.89 KB

The patch in #1 broke the item xpath for me when the document uses a default namespace, such as in Google's KML files:

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://earth.google.com/kml/2.2">
<Document>
...
// A whole bunch of tags without a namespace prefix.
...

I explicitly registered the namespace and assigned it a prefix, which is needed becuase SimpleXML doesn't do default namespaces, and I used that prefix in the item and item-id xpaths. The setup code was almost identical to that of the OP in #1290706: MigrateItemsXML: Centralize loading of XML.

It did work before this change because my xpaths were simple enough to be concatenated, as demonstrated in the OP, and Migrate did not run another XPath query on the element(s) returned from a previous query.

As noted in the SimpleXMLElement::registerXPathNamespace documentation, that method only "creates a prefix/ns context for the next XPath query".
The namespaces need to be re-registered on each element returned from an XPath query.

In my use case, the problem first emerged in the below snippet, when fetching a list of ids to count the records to import.
It always returned an empty result because $xml contained a SimpleXMLElement gotten from a previous XPath query, and the namespaces were no longer registered. ($xml still keeps track of the default namespace, and $this->itemXpath did contain the prefix, but the prefix was no longer registered to that namespace.)

+++ b/plugins/sources/xml.incundefined
@@ -437,18 +437,16 @@ class MigrateItemsXML extends MigrateItems {
+    $result = $xml->xpath($this->itemXpath);
 
-    $result = $xml->xpath($full_xpath);
+    $ids = array();
     if ($result) {
-      if (count($result) > 1) {
-        foreach ($result as $id) {
+      foreach ($result as $element) {

My solution was to allow the creator of the MigrateItemsXML class to pass in a list of prefix/namespaces that should be automatically re-registered on the elements returned from an XPath query, as shown in the patch. I can now use my registered prefix like this:

    $items_url = $xml_folder . 'mapping-data.kml';
    $item_xpath = '//k:Document/k:Placemark';
    $item_ID_xpath = 'k:name';
    $items_class = new MigrateItemsXML($items_url, $item_xpath, $item_ID_xpath, array('k' => 'http://earth.google.com/kml/2.2'));

Note: I've not looked at if something needs to be done about the class(es) handling sources which do not have the item and id together.

mikeryan’s picture

Status: Needs review » Closed (fixed)

Please don't reopen long-closed issues, particularly when you're proposing a complex patch that's only tangentially related - please open a fresh issue for handling namespaces in MigrateListXML/MigrateItemsXML.

windmaomao’s picture

#11 can address answers on https://drupal.org/node/1961316

I'll try #11's solution, basically i have the same issue.