Migrations don't always finish [#2829238]

Problem/Motivation

We running migrations if there's not enough memory to compete the migration it'll not complete and you won't know because the source count is not displayed.

Proposed resolution

Run migrations in batches until they are complete.

Remaining tasks

User interface changes

None

API changes

None

Data model changes

None

Comment	File	Size	Author
#2	2829238-2.patch	3.14 KB	alexpott

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

21 November 2016 at 09:31

alexpott created an issue. See original summary.

Comment #2

alexpott

he/they

English

🇪🇺🌍

CreditAttribution: alexpott at Chapter Three commented 21 November 2016 at 09:32

Status:

Active

» Needs review

File	Size
2829238-2.patch	3.14 KB

Comment #3

dawehner

German

CreditAttribution: dawehner at Chapter Three commented 21 November 2016 at 11:13

+++ b/migrate_drush.drush.inc
@@ -63,10 +64,66 @@ function drush_migrate_drush_run($migrations) {
+    $source_count = $migration->getSourcePlugin()->count();
+    if ($source_count > 0) {
+      $context['finished'] = $id_map->processedCount() / $source_count;
+    }
+    else {
+      $context['finished'] = 1;
+    }

The original code in MigrateExecuteable uses

    while ($source->valid()) {
      $row = $source->current();
      $this->sourceIdValues = $row->getSourceIdValues();

      try {
        $this->processRow($row);
        $save = TRUE;
      }

The code example here would directly stop for migrations, which cannot be counted. I guess though at least generator based once would work though, see #2684567: Requiring a migration w/ a source plugin using a generator fatals.
Maybe calling out to valid would be everything which is needed.

Comment #4

alexpott

he/they

English

🇪🇺🌍

CreditAttribution: alexpott at Chapter Three commented 21 November 2016 at 11:58

Maybe it is best to use allRowsProcessed() because that's how migrate checks is something has run...

But that does:

  public function allRowsProcessed() {
    $source_count = $this->getSourcePlugin()->count();
    // If the source is uncountable, we have no way of knowing if it's
    // complete, so stipulate that it is.
    if ($source_count < 0) {
      return TRUE;
    }
    $processed_count = $this->getIdMap()->processedCount();
    // We don't use == because in some circumstances (like unresolved stubs
    // being created), the processed count may be higher than the available
    // source rows.
    return $source_count <= $processed_count;
  }

Which is why I went the way I did.

Comment #5

benjy CreditAttribution: benjy commented 24 November 2016 at 11:26

I tested this in place of my other patch which was just setting memory to -1 and it blows up in our CI environment, not sure why but something changes.

Also noticed that the output is much more verbose, I used to get one output at the end of the migration, now i'm getting hundreds of lines of output:

_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]
lg_files processed 403 out of 409 STATUS_IMPORTED 243 STATUS_NEEDS_UPDATE 0 STATUS_FAILED 160              [ok]

Comment #6

benjy CreditAttribution: benjy commented 24 November 2016 at 11:34

Looks like something with this patches causes it to get stuck in an infinite loop for me. Could be because I have a few failed rows, not looked any further.