Hi guys,

When I import a XML with 60k rows import speed is very slow, around 15/min
When I import a XML in same migration with 1k rows, speed is quick, 130+/min
Why rows count effect to speed?
How to fix it?

Thx.

Comments

simon.westyn’s picture

simon.westyn commented

From my experience you should use MigrateSourceMultiItems for small xml files, and MigrateSourceXML for large files.

Small xml:
$this->source = new MigrateSourceMultiItems($items_class, $fields);

Big xml:
$this->source = new MigrateSourceXML('ftp://<username>:<password>@<ipaddress>/file.xml', $item_xpath, $item_ID_xpath);

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Issue summary: View changes
Priority: Major » Normal
Status: Active » Fixed

To expand on Simon's reply - MigrateSourceMultiItems() uses the SimpleXML PHP extension, which reads and parses the entire XML file into memory. The advantage of using this is you have the full power of xpaths at your disposal, to instantly address any data within the XML, but the disadvantage is memory and speed for large files. MigrateSourceXML uses the XMLReader extension, which allows it to iteratively parse and process the XML file, but the disadvantage is more limited xpath support. See https://drupal.org/node/1152156 for more information.

See also https://drupal.org/node/2136603 for general help diagnosing and improving performance issues.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.