Out of Memory on Initial Select call [#2301565]

Hi,

I'm looking for ideas on how to go about solving this. I'm trying to migrate files from a D6 DB to D7 (via drush). I'm getting an out of memory situation on the initial select statement because it's trying to slurp 4.3M rows all at once (it's an old site):

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 32 bytes) in .../includes/database/database.inc on line 2171
...
PHP  15. DatabaseConnection->query($query = 'SELECT f.*\nFROM \n{files} f\nORDER BY timestamp ASC', $args = array (), $options = array ('return' => 1)
) .../includes/database/select.inc:1264

Yeah, my php.ini memory_limit in my dev environment is 1G. That's too high for me to run it on the machine where the actual migration is going to run as it is, so I don't want to try and keep raising it until it works. I tried '--limit="100 rows",' but that's clearly not being passed to the query (in any form, I think I understand why). I'm using the stock DrupalFile6Migration, version 7.x-2.1-beta1+28-dev.

My thinking is that I need a LIMIT clause on the SELECT statement. I'm reviewing the migrate module to see if something like that is supported. To try and come up with a way to "batch" selects and do some number at-a-time. Select 10000 rows and override MigrateSource::getNextRow() to select the next set of 10000 when it runs out of rows.

Any suggestions?

Finally, would this question more suitable to the migrate module's issue queue?

TIA,
Dave

Comment	File	Size	Author
#2	file.zip	1.26 KB	dnmurray

Comments

Comment #1

dnmurray commented 11 July 2014 at 12:14

Issue summary:

View changes

edit: typos.

Comment #2

dnmurray commented 11 July 2014 at 18:55

Status	File	Size
new	file.zip	1.26 KB

The attachment seems to do the trick. Invoked with the following code in my migrate module:

  $arguments = $common_arguments + array(
    'description' => t('Migration of files from Drupal 6'),
    'machine_name' => 'File',
    'user_migration' => 'User',
    'dependencies' => array('User',),
    'batch_size' => 10000,
  );
  Migration::registerMigration('DrupalFile6MigrationBatched', $arguments['machine_name'],
                               $arguments);