Currently the installer just gets a list of all available migrations and then does $migration->processImport($options);
However, processImport() returns a value, and if that value is MigrationBase::RESULT_INCOMPLETE, then the migration should be run again.
Our code currently doesn't do that, which means that when the system is low on memory, the migration completes halfway through and the installation is completed without the complete set of content.

We need to rewrite our batch function to rerun the migration in that case. We also need special code for drush (since the batch doesn't help there), similar to the one in

Members fund testing for the Drupal project. Drupal Association Learn more


bojanz’s picture

Issue tags: -RC blocker +RC2 blocker

This can wait until RC2

vasike’s picture

Status: Active » Needs review

there's commit for batch function for kickstart migration to re-run for incomplete migration

based on migrate_ui_batch() function code.

to do : special code for drush

jsacksick’s picture

joshmiller’s picture

Did this get committed and not updated? It's the only 2.0 release blocker that wasn't "complete (fixed)" ?

Curious because I'm going through all the drush related issues in the que and this one seemed like it needed and update (10 weeks old).

nedjo’s picture

Given this is a processing-heavy operation, there may be a role here for drupal_set_time_limit(). I've opened a related issue on migrate: #1892296: Allocate sufficient time for migration processes.

nedjo’s picture

Re #5. As Moshe pointed out in #1892296, batch API takes care of setting the time limit.

lsolesen’s picture

Issue summary: View changes

@joshmiller @bojanz Has this been fixed? It is tagged 2.0 release blocker but still needs review :)

bojanz’s picture

No, this was never committed :/ Needs to be tested in a low memory situation (get the system to swap) to confirm that it retries / continues where the previous code doesn't.

Related bug report which might be useful in the future: #1851806: Install slow on import content

mglaman’s picture

Status: Needs review » Active

No patch, putting back to "active" for now

GuyPaddock’s picture

Try this one. This is taken from the migration code we use in all of our installation profiles, which in turn was originally based on Commerce Kickstart's migration code, so this has come full circle. :)

GuyPaddock’s picture

Here's one that will pass Drupal conventions for i18n.

(Our original code has watchdog statements in this function as well, and they share the same messages as the ones we pass through t() for the Batch API, so that's why my original patch puts the strings in a variable. As you know, watchdog requires the untranslated version.)

mglaman’s picture

Looks good to me, tested it out. The code follows the logic that Migrate provides for this use case. I'm all for getting it in pending bojanz's thoughts per #8 (testing in low memory environment)