Problem/Motivation

hooks--
events++

SourcePluginBase::prepareRow() exists to provide an application-level opportunity to manipulate a source row, and potentially decide based on its contents to ignore it entirely, before entering the processing pipeline. This has been incubating in Migrate Plus for 3+ years. With us moving things from migrate contrib into Drush core, it is more important to have an event to handle this processing.

Proposed resolution

SourcePluginBase::prepareRow() should dispatch a PREPARE_ROW event instead of invoking hooks.

const PREPARE_ROW = 'migrate.prepare_row';

Remaining tasks

SourcePluginBase::prepareRow() should call PREPARE_ROW in addition to the hook.
If possible, trigger an error/warning when the older hook is utilized.

User interface changes

N/A

API changes

hook_migrate_prepare_row and hook_migrate_MIGRATION_ID_prepare_row deprecated.
const PREPARE_ROW = 'migrate.prepare_row'; added

Comments

mikeryan’s picture

Issue summary: View changes
mikeryan’s picture

Status: Active » Needs review
StatusFileSize
new7.27 KB
benjy’s picture

SourcePluginBase::prepareRow() should invoke $this->migration->prepareRow() instead of hooks.

That means to use this in the migration you need to hook_entity_info_alter() and extend the migration entity? I don't think we want that.

chx’s picture

> if you were to update a typical D7 migration module to D8 converting field mappings etc. to configuration, almost all of the remaining necessary PHP code would be prepareRow() implementations.

No way! It would be in process plugins. prepare row is not supposed to do much. It is supposed to help creating a "canonical" representation of the source object whatever it is. Once that's done, it's game over. I hope that hook will be rarely used.

mikeryan’s picture

Status: Needs review » Needs work

Right - my pattern indicates D7-dimensional thinking, where everything was done by extending the Migration class... Yes, that's not the place to do the work prepareRow() has traditionally done.

But, that work still is necessary. Process plugins only deal with one field in isolation - they're the equivalent of callbacks in the D7 Migrate. We still need prepareRow() - a place for application-level code to hook in and help generate the canonical source data. The three things that typically happened in prepareRow() still need to happen:

  1. Adding data to the row - most typically with database sources, multi-valued related tables that if incorporated directly in the base query would cause it to return multiple rows for a given source item.
  2. Manipulation of the data that involves more than a single field - computing a value based on multiple source fields, combining multiple fields into one or breaking up a field into multiple fields, etc. Yes, process plugins have access to the Row, but I feel in many of these cases it's better to manipulate and create the canonical Row up front than to do it in the context of a single field.
  3. Deciding based on the row data to ignore a given row (as opposed to aborting it due to an error later).

And, of course, we currently have a mechanism to do this - my objection is that the mechanism should be object-oriented instead of hook-based, and the ignore signal should be an exception rather than a boolean return value.

So, taking a minute to think it through this time... The logical place to do this is of course the source plugin, which is more easily (and often) extended than the migration class, and whose responsibility is handling that row. As far as manipulating the row goes, all you need to do is extend prepareRow(). The only issue then is dealing with the ignore case, and I think that's best handled by moving the logic (writing the id mapping etc.) to MigrateExecutable upon catching MigrateSkipRowException. Side benefit - that also addresses #2443617: Find a way for SourcePluginBase to get MigrateExecutable, eliminating any need for the source plugin to know anything about the executable class.

mikeryan’s picture

Status: Needs work » Needs review
StatusFileSize
new20.43 KB

OK, removing the hooks and using exceptions to skip rows...

benjy’s picture

I'm not sure I agree with removing the hook unless we have something nearly as easy without having to replace the entire source.

Also, I don't see the benefit of the exception instead of a return value when it is caught again in the same class or the parent? It would have made more sense if the exception propagated to the executable. Would that solve our issue with the source currently needing the executable for the saveMapping on skipped rows?

chx’s picture

> Would that solve our issue with the source currently needing the executable for the saveMapping on skipped rows?

Just to make sure we see clearly,

      $this->migrateExecutable->saveQueuedMessages();
      $id_map->saveIdMapping($row, array(), MigrateIdMapInterface::STATUS_IGNORED, $this->migrateExecutable->rollbackAction);

this is all we use the executable in the source base ; id_map is from the entity itself. But this is material for #2443617: Find a way for SourcePluginBase to get MigrateExecutable

mikeryan’s picture

Status: Needs review » Needs work
Related issues: +#2443617: Find a way for SourcePluginBase to get MigrateExecutable

To me, coming from an OO point of view, overriding a source class to alter the source row seems easier and more natural than implementing a hook. Certainly when you're overriding the class anyway to define a query, as with basically all the migrate_drupal sources, it makes much more sense to me to have all the code for that source together in one class rather than split between the class and a hook. And migrate_drupal is already overriding prepareRow() to add source data in several source plugins.

I did get a little lazy with the exception by catching it in next() - in the executable the logic gets more complex because you have to catch it from both the rewind() (which calls next()) and next() calls. But, it is worth it to also take care of #2443617: Find a way for SourcePluginBase to get MigrateExecutable (and also will make it easier to track the 'skipped' counter in #2443081: Make MigrateExecutable statistics publicly available). I'll reroll it that way.

mikeryan’s picture

Adding #2443081: Make MigrateExecutable statistics publicly available as a related issue - whichever of these two issues gets committed first, the other will need a reroll.

mikeryan’s picture

Ah, now I remember why I was catching SkipRow in next() - because next() needs to leave the iterator pointing at a valid row unless it's reached the end. If, say, the 2nd row of a 10-row data set needs to be skipped, we throw the exception and MigrateExecutable catches it at the end of the import() loop - then $source->valid() fails and we drop out of the loop without processing the last 8 rows. I suppose MigrateExecutable on catching the SkipRow could then loop until it gets a row that does not throw SkipRow... But of course that's really the iterator's job, to find the next suitable row for migration. So, I do think next() is the right place, and we'll need to separately address the references to MigrateExecutable from SourcePluginBase.

Just to be clear, I do still think exceptions are better than propagating booleans up the call chain. I've become really tired of

public function prepareRow($row) {
  if (parent::prepareRow($row) === FALSE) {
    return FALSE;
  }
  ...

in every D7 migration. prepareRow() will be used less frequently in D8, but it's still a clunky approach.

Meanwhile, looking at SourcePluginBase::prepareRow(), all it's doing now (other than statistics tracking that will be obsoleted by #2443081: Make MigrateExecutable statistics publicly available) is handling trackChanges. This is something that needs to be done after all row preparation is done, but there's nothing to guarantee specific source plugins call parent::prepareRow() last. And typically, at least as used in D7, if you have an intermediate class that you extend, you want to call the intermediate class's prepareRow() first, not last. So, I think the trackChanges handling should be taken out of prepareRow() and be done in next() after the call to prepareRow().

Is it too late to replace calling prepareRow() inline with event listeners? (/me isn't quite sure how serious he is...)

benjy’s picture

I don't see much benefit in having an exception vs a return boolean if it just to be caught in a parent implementation. However, I don't think it's too late to add an event if we think that's the best approach, it would make more sense to catch an exception that was thrown by an event listener.

mikeryan’s picture

Status: Needs work » Postponed

I'm becoming more enamored of the event approach - let's see how #2535458: Dispatch events at key points during migration goes and if it succeeds look at implementing prepareRow as an event as well.

mikeryan’s picture

Just a note that adding a prepareRow event should allow contrib to handle idlist filtering entirely on its own, so we'll be able to remove that stuff from core: #2522012: Remove broken idlist handling, replace with more robust exception handling .

mikeryan’s picture

Title: Refactor prepareRow() to use methods and exceptions » Refactor prepareRow() to use events and exceptions
Issue summary: View changes
mikeryan’s picture

Status: Postponed » Needs review
StatusFileSize
new37.38 KB

Kinda feeling like I'm tilting at windmills here - this wouldn't be bad if it were designed in from the start, but I fear it's going to be too disruptive to make the change at this point. Submitting what I have of the patch for testbot, but I didn't get to changing the existing prepareRow() implementations into event listeners.

benjy’s picture

but I didn't get to changing the existing prepareRow() implementations into event listeners.

I don't think we should ever do this. If I'm a source plugin, I have a prepareRow() method that I can overload and just call my parent, I shouldn't have to implement an event listener to do stuff there, that's for others that want to extend/change/react based on what the default source does.

mikeryan’s picture

So, I'm going to back off a bit on this and take it in pieces. My immediate goal is to be able to implement the idlist functionality entirely on the frontend where it belongs, and remove the non-functional idlist stuff from core. The first stage of my plan is to do the core-side parts of that in #2522012: Remove broken idlist handling, replace with more robust exception handling (hint: I expect it will involve exceptions in prepareRow()). Once that gets into core, then we can address making prepareRow() an event in core here in this issue.

mgifford’s picture

Status: Postponed » Needs review

I think that blocker's gone now.

Status: Needs review » Needs work

The last submitted patch, 16: refactor_preparerow-2488836-16.patch, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

mikeryan’s picture

Version: 8.1.x-dev » 8.2.x-dev
mikeryan’s picture

Version: 8.2.x-dev » 9.x-dev
Status: Needs work » Postponed
Issue tags: -neworleans2016, -Migrate BC break

Realistically, we aren't going to redesign prepareRow() at this stage - let's look at it again in 9.x.

catch’s picture

Version: 9.x-dev » 8.3.x-dev
Priority: Normal » Minor

If wanted to add the event, we could do that in 8.3.x (doesn't mean we have to of course, and if it's just hooks vs. events I personally prefer #2237831: Allow module services to specify hooks for that). Bumping down to 8.3.x for that reason, but moving to minor since I don't think this is enabling anything new? The return change seems like the smallest difference here.

OnkelTem’s picture

Sorry I haven't read the whole discussion just having a comment on this and one idea to share (which I've already implemeted):

> SourcePluginBase::prepareRow() should dispatch a PREPARE_ROW event instead of invoking hooks.

Not only that. It should allow to override in child classes. I don't like EVENT system here at all because
1) event handlers are totally unconnected from the migration
2) get called on every sneeze! I.e. there are always multituide of empty calls
3) from (1) -> you have to write if-s to just filter out unnecessary events, not related to your migration.

That said, here is a solution:

1) add checkRow() method to SourcePluginBase:

  /**
   * Checks row data and skips the row in case of errors.
   *
   * It is similar to prepareRow() but can skip rows and save them to map-table.
   * It can also throw MigrateSkipRowException.
   *
   * @param \Drupal\Migrate\Row $row
   *   The row object.
   *
   * @throws \Drupal\migrate\MigrateSkipRowException
   *
   * @return bool|NULL
   *   FALSE if this row needs to be skipped.
   */
  public function checkRow(Row $row) { }

2) rewrite prepareRow like this:

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {
    $result = TRUE;
    try {
      // here should go a hook-event part, I'm skipping it for now
      // ...
      // ...
      $result = $this->checkRow($row); /// <<< NEW LINE
      $skip = $result === FALSE; /// <<< NEW LINE
      $save_to_map = TRUE;
    }
    catch (MigrateSkipRowException $e) {
      $skip = TRUE;
      $save_to_map = $e->getSaveToMap();
      if ($message = trim($e->getMessage())) {
        $this->idMap->saveMessage($row->getSourceIdValues(), $message, MigrationInterface::MESSAGE_INFORMATIONAL);
      }
    }

    // We're explicitly skipping this row - keep track in the map table.
    if ($skip) {
      // Make sure we replace any previous messages for this item with any
      // new ones.
      if ($save_to_map) {
        $this->idMap->saveIdMapping($row, array(), MigrateIdMapInterface::STATUS_IGNORED);
        $this->currentRow = NULL;
        $this->currentSourceIds = NULL;
      }
      $result = FALSE;
    }
    elseif ($this->trackChanges) {
      // When tracking changed data, We want to quietly skip (rather than
      // "ignore") rows with changes. The caller needs to make that decision,
      // so we need to provide them with the necessary information (before and
      // after hashes).
      $row->rehash();
    }
    return $result;
  }
}

Now in any Source child we can easily do (example):

  /**
   * {@inheritdoc}
   */
  public function checkRow(Row $row) {
    // Prepare `destination_file_uri` source property for `entity:file` destination plugin
    // We are mirroring remote directory structure
    $local_file_uri = NULL;
    if ($value = $row->getSourceProperty('image_path')) {
      // We support only local paths starting with "/".
      if (strpos($value, '/') === 0) {
        $path = substr($value, 1);
        $local_file_uri = $this->get('local_files_uri') . '/' . $path;
      }
      else {
        throw new MigrateSkipRowException(t('Not handled file uri: @uri', ['%uri' => $value]));
      }
    }
    $row->setSourceProperty('destination_file_uri', $local_file_uri);
  }

OR... we could just return FALSE.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

joachim’s picture

Priority: Minor » Normal
Status: Postponed » Active
Issue tags: +Needs issue rescope

> SourcePluginBase::prepareRow() should dispatch a PREPARE_ROW event instead of invoking hooks.
> Catch MigrateSkipRowException in SourcePluginBase instead of relying on a FALSE return.

That's two changes, not one. This should be split into two issues.

> Catch MigrateSkipRowException in SourcePluginBase instead of relying on a FALSE return.

That's a small, easy change, and an improvement to DX, as returning FALSE to skip means that developers have to figure out what to return if you don't want to skip. I've seen lots of migration source plugins that return TRUE here to mean the row is ok, when all they need to do is fall out of the method.

The current behaviour of relying on a FALSE return can be kept for BC.

heddn’s picture

Perhaps we should re-boot the conversation here on adding an event. See https://github.com/drush-ops/drush/pull/3402 where we are trying to bring things from migrate_run (migrate_tools fork) into drush core. In there, we kinda need an event unless we do really hacky things.

claudiu.cristea’s picture

@joachim, I split the event constant, event class and event dispatching in #2952291: Dispatch the PREPARE_ROW event. Deprecate prepare_row_alter hooks.

heddn’s picture

Issue summary: View changes
Status: Active » Needs work
Issue tags: -Needs issue rescope +Needs change record, +Contributed project blocker

Re-worked the IS to remove the try/catch stuff. If we want to handle that, we can open that in a new issue. This is now only focused on the MigratePrepareRowEvent moving from migrate_plus into core.

heddn’s picture

Title: Refactor prepareRow() to use events and exceptions » Refactor prepareRow() to add events in addition to hook
claudiu.cristea’s picture

mikeryan’s picture

Status: Needs work » Closed (duplicate)

The scope here now appears identical to #2952291: Dispatch the PREPARE_ROW event. Deprecate prepare_row_alter hooks, so this seems redundant - we might as well close this in favor of the one with the active patch. I'm not particularly inclined to pursue the exception handling changes at this point, but if someone else does they can open a new issue for that.