Cannot use Source Plugin that uses Generators with import sync

NOTE: I am not suggesting the following as a solution!

For reference, if I modify migrate's SourcePluginBase to add the following function:

  public function resetIterator() {
    $this->iterator = $this->initializeIterator();
  }

And then modify MigrationImportSync::sync() to add a single line at the end of the if loop:

      $this->dispatcher->dispatch(MigrateEvents::POST_ROLLBACK, new MigrateRollbackEvent($migration));
      $source->resetIterator();

      $id_map->prepareUpdate();
      $source = clone $migration->getSourcePlugin();
      $source->rewind();

then migrate import with sync works exactly as expected.

Aha! I found another solution:

If instead I swap out the following line in MigrationImportSync::sync():

Replace:

      $id_map->prepareUpdate();
      $source = $migration->getSourcePlugin();
      $source->rewind();

With:

      $id_map->prepareUpdate();
      $source = clone $migration->getSourcePlugin();
      $source->rewind();

and ensure to run drush cim --sync with "--skip-progress-bar", it also works. The issue here is that MigrateDrushCommandProgress::initializeProgress calls count(), which initializes the generator early. If the progress bar is skipped, the generator isn't initialized until sync is actually run.

I don't suggest this as a proper solution, but it *does* work.

I've spent yesterday afternoon and this morning figuring out PHPUnit and comparing my migrate-import script with migrate_tools' DrushCommandsTest.

Without my patch, for some reason, the migrate_tools test is just fine, but my migrate import using drush and migrate_source_csv fails.

DrushCommandsTest's version seems to instantiate a new MigrateExecutable between the sync and the import (creating a new Generator), where as my drush import csv doesn't.

Logic suggests it's either a migrate_source_csv issue, or a user error of some kind, but I'm not sure which.

I'm going to continue to investigate.

OK, I understand the problem at least now. I'm using two different SourcePlugins.

DrushCommandsTest uses an EmbeddedDataSource, as defined in migrate_plus.migration.fruit_terms.yml.

EmbeddedDataSource::initializeIterator() is defined as:

  public function initializeIterator() {
    return new \ArrayIterator($this->dataRows);
  }

However, my CSV code is using migrate_source_csv's CSV source plugin. There, CSV::initializeIterator() is defined as:

  public function initializeIterator() {
    $header = $this->getReader()->getHeader();
    if ($this->configuration['fields']) {
      // If there is no header record, we need to flip description and name so
      // the name becomes the header record.
      $header = array_flip($this->fields());
    }
    return $this->getGenerator($this->getReader()->getRecords($header));
  }

In short, EmbeddedDataSource uses an ArrayIterator. CSV uses a Generator iterator. Generators cannot be rewound, but ArrayIterators can.

It all makes sense, migrate_source_csv likely did this to save memory on large CSV imports.

I'm going to update the ticket title and description. Not clear where to proceed from here.

@heddn - is this a migrate_source_csv issue, or a migrate_tools issue?

Just a note that migrate_source_csv:3.1 and migrate_source_csv's current dev branch still both use generators instead of simple iterators. This isn't as simple as switching migrate_source_csv to ^3.