I've got what I thought was going to be a fairly straightforward migration of a very large data set (~10M rows of .csv supplied in 9 files) but when I run one (on a Macbook with MAMP) it took 21.5 hours processing 650 lines/second to run the first file of ~1M rows. I would really appreciate advice from anyone about where to start in terms of improving the performance. (And if I'm barking up the wrong tree and there's not much to be done, please let me know before I waste too much time fiddling and just learn to be patient)
The csv data is in a bit of a screwey format in that I have a row which contains the 'customer' data which gets turned into 'customer' nodes (that migrates fairly quickly) and 'rate analysis' data which gets turned into a field collection as part of the subsequent migration, followed by 10-14 rows of bill period data which gets migrated into a field collection. Then there's another customer & rate row and another 10-14 rows of bill period data.
So I wrote this:
function migrate_customers_migrate_api() {
$api = array(
'api' => 2,
'migrations' => array(
'CustomerData1' => array('class_name' => 'CustomerDataMigration1'),
'RateAnalysis1' => array('class_name' => 'RateAnalysisMigration1'),
'BillPeriod1' => array('class_name' => 'BillPeriodMigration1'),
),
);
return $api;
}
//then there's the CustomerDataMigration
//then there's the RateAnalysisMigration which works fast enough too
/**
* Migrate the bill period data into the entity collections for each rate analysis.
*/
abstract class BillPeriodMigration extends DynamicMigration {
public function __construct() {
parent::__construct();
$this->description = t('Import bill period data from CSV file.');
//$this->softDependencies = array('CustomerData');
// Create a map object for tracking the relationships between source rows
$this->map = new MigrateSQLMap($this->machineName,
array(
'account_ID' => array('type' => 'varchar',
'length' => 12,
'not null' => TRUE,
'description' => 'account_ID',
),
'rate_plan_id' => array('type' => 'varchar',
'length' => 255,
'not null' => TRUE,
'description' => 'rate_plan_id',
),
'bill_period_start_date' => array('type' => 'varchar',
'length' => 255,
'not null' => TRUE,
'description' => 'bill_period_start_date',
),
),
MigrateDestinationNode::getKeySchema()
);
$this->destination = new MigrateDestinationFieldCollection(
'field_bill_period',
array('host_entity_type' => 'node')
);
//$this->addFieldMapping('host_entity_id', 'account_ID')->sourceMigration('CustomerData');
$this->addFieldMapping('field_rate_plan_id', 'rate_plan_id');
$this->addFieldMapping('field_bill_period_cost', 'bill_period_cost');
$this->addFieldMapping('field_bill_period_start', 'bill_period_start_date');
$this->addFieldMapping('field_bill_period_end', 'bill_period_end_date');
$this->addFieldMapping(NULL, 'account_ID');
$this->addUnmigratedSources(array(
'analysis_date',
'util_internal_id',
'customer_first_name',
'customer_last_name',
'record_type',
"mailing_address_line_1",
"mailing_address_line_2",
"mailing_address_line_3",
"mailing_address_line_4",
"service_address_line_1",
"service_address_line_2",
"service_address_line_3",
"service_address_line_4",
'default_plan',
'current_plan',
'lowest_total_cost_plan',
'total_cost',
'total_start_date',
'total_end_date',
'total_bill_periods',
'has_errors'
));
}
function prepareRow($row) {
if (parent::prepareRow($row) === FALSE) {
return FALSE;
}
// Don't import customer rows or glitched bill periods with no cost
if($row->record_type == "TOTAL" || $row->bill_period_cost == ''){
return FALSE;
}
}
}
//BILL PERIOD MIGRATIONS
class BillPeriodMigration1 extends BillPeriodMigration {
public function __construct() {
// Always call the parent constructor first for basic setup
parent::__construct();
$file = dirname(__FILE__) . '/data/'.FILE1;
//$file = dirname(__FILE__) . '/data/2013_Feb_OPower_BRATInput_1237568810.csv';
$this->source = new MigrateSourceCSV($file, array(), array('header_rows' => 1) );
$this->addFieldMapping('host_entity_id', 'account_ID')->sourceMigration('CustomerData1');
}
}
Comments
Comment #1
mikeryanAs a general rule, the bottleneck in migration is writing to the Drupal database, so optimizing the MySQL server for writing can help a lot.
In your particular case, field_collection is a major bottleneck. The problem is that collection fields and their host entities are double-linked - each points to the other, which presents a chicken-and-egg problem: you have to migrate one, then the other, then go back and point the first one at the second one. The field collection handler is set up so you need to migrate the host entity first. The handler itself then, once it migrates the field collections, must load and save the host entity - i.e., in the full migration, all the host entities are saved at least twice, once on initial migration and again by each collection field. This is, obviously, a lot of overhead with your volume.
In your case, it may be worthwhile to hack the field handler to write the host entity's field table directly with a pointer to the collection, instead of loading/saving it. The risk here is if there are any hooks that need to fire when the host entity is saved dependent on that collection field, but in most cases you should be able to get away with this.
Comment #2
cduwe commentedThanks for the update. In my particular case the solution was to go back to the client and tell them there would be a delay and then for all of us to be patient. By not hacking anything too much I benefitted by being able to stop, restart, etc. my migrations. A week later and it appears we got all that data in there correctly. Thanks again for the module.
Comment #3
Dru18 commentedEvidently field collection have a seriously performance issue by design. I hope this flaw will disappear with the impending Drupal 8 release. Meanwhile, I have to deal with it in the migration. In my case, I have to take care of field collections, term references, entity references and media field; all are attached to a node. Some field collections also have address filed and entity references additionally. The migration process has to accept on-going data. It finds node id and plug the data across the fields by reading JSON structure.
Entity references and term references work fine w/o any issue. But the field collection CRUD is very expensive. If there're more than five inserts/updates, the migration struggles a lot. Originally I broke them down to each FC migration process. This ended up becoming an inefficient route because I am importing data through JSON feed; which means I have to make redundant calls as many as the field collection count. So I decided to consolidate them and created a process per node. If a node has a short field list, it is tolerable (less than a second). Fortunately, however, one node has a field that has a long list of field collections attached; have to update each one of them in case there's any record change. The bottom line is processing one node takes about 5-6 seconds if this is an initial run. In my case, running update doesn't make any difference since I have to update (the same as creating). I may be able to directly attache FC data to a node according to Mike's suggestion so that I can avoid the FC wrappers which has a huge performance drawback. Frustration rules.