Problem/Motivation
hooks--
events++
SourcePluginBase::prepareRow() exists to provide an application-level opportunity to manipulate a source row, and potentially decide based on its contents to ignore it entirely, before entering the processing pipeline. This has been incubating in Migrate Plus for 3+ years. With us moving things from migrate contrib into Drush core, it is more important to have an event to handle this processing.
Proposed resolution
SourcePluginBase::prepareRow() should dispatch a PREPARE_ROW event instead of invoking hooks.
const PREPARE_ROW = 'migrate.prepare_row';
Remaining tasks
SourcePluginBase::prepareRow() should call PREPARE_ROW in addition to the hook.
If possible, trigger an error/warning when the older hook is utilized.
User interface changes
N/A
API changes
hook_migrate_prepare_row and hook_migrate_MIGRATION_ID_prepare_row deprecated.
const PREPARE_ROW = 'migrate.prepare_row'; added
| Comment | File | Size | Author |
|---|---|---|---|
| #16 | refactor_preparerow-2488836-16.patch | 37.38 KB | mikeryan |
| #6 | refactor_preparerow-2488836-6.patch | 20.43 KB | mikeryan |
Comments
Comment #1
mikeryanComment #2
mikeryanComment #3
benjy commentedThat means to use this in the migration you need to hook_entity_info_alter() and extend the migration entity? I don't think we want that.
Comment #4
chx commented> if you were to update a typical D7 migration module to D8 converting field mappings etc. to configuration, almost all of the remaining necessary PHP code would be prepareRow() implementations.
No way! It would be in process plugins. prepare row is not supposed to do much. It is supposed to help creating a "canonical" representation of the source object whatever it is. Once that's done, it's game over. I hope that hook will be rarely used.
Comment #5
mikeryanRight - my pattern indicates D7-dimensional thinking, where everything was done by extending the Migration class... Yes, that's not the place to do the work prepareRow() has traditionally done.
But, that work still is necessary. Process plugins only deal with one field in isolation - they're the equivalent of callbacks in the D7 Migrate. We still need prepareRow() - a place for application-level code to hook in and help generate the canonical source data. The three things that typically happened in prepareRow() still need to happen:
And, of course, we currently have a mechanism to do this - my objection is that the mechanism should be object-oriented instead of hook-based, and the ignore signal should be an exception rather than a boolean return value.
So, taking a minute to think it through this time... The logical place to do this is of course the source plugin, which is more easily (and often) extended than the migration class, and whose responsibility is handling that row. As far as manipulating the row goes, all you need to do is extend prepareRow(). The only issue then is dealing with the ignore case, and I think that's best handled by moving the logic (writing the id mapping etc.) to MigrateExecutable upon catching MigrateSkipRowException. Side benefit - that also addresses #2443617: Find a way for SourcePluginBase to get MigrateExecutable, eliminating any need for the source plugin to know anything about the executable class.
Comment #6
mikeryanOK, removing the hooks and using exceptions to skip rows...
Comment #7
benjy commentedI'm not sure I agree with removing the hook unless we have something nearly as easy without having to replace the entire source.
Also, I don't see the benefit of the exception instead of a return value when it is caught again in the same class or the parent? It would have made more sense if the exception propagated to the executable. Would that solve our issue with the source currently needing the executable for the saveMapping on skipped rows?
Comment #8
chx commented> Would that solve our issue with the source currently needing the executable for the saveMapping on skipped rows?
Just to make sure we see clearly,
this is all we use the executable in the source base ; id_map is from the entity itself. But this is material for #2443617: Find a way for SourcePluginBase to get MigrateExecutable
Comment #9
mikeryanTo me, coming from an OO point of view, overriding a source class to alter the source row seems easier and more natural than implementing a hook. Certainly when you're overriding the class anyway to define a query, as with basically all the migrate_drupal sources, it makes much more sense to me to have all the code for that source together in one class rather than split between the class and a hook. And migrate_drupal is already overriding prepareRow() to add source data in several source plugins.
I did get a little lazy with the exception by catching it in next() - in the executable the logic gets more complex because you have to catch it from both the rewind() (which calls next()) and next() calls. But, it is worth it to also take care of #2443617: Find a way for SourcePluginBase to get MigrateExecutable (and also will make it easier to track the 'skipped' counter in #2443081: Make MigrateExecutable statistics publicly available). I'll reroll it that way.
Comment #10
mikeryanAdding #2443081: Make MigrateExecutable statistics publicly available as a related issue - whichever of these two issues gets committed first, the other will need a reroll.
Comment #11
mikeryanAh, now I remember why I was catching SkipRow in next() - because next() needs to leave the iterator pointing at a valid row unless it's reached the end. If, say, the 2nd row of a 10-row data set needs to be skipped, we throw the exception and MigrateExecutable catches it at the end of the import() loop - then $source->valid() fails and we drop out of the loop without processing the last 8 rows. I suppose MigrateExecutable on catching the SkipRow could then loop until it gets a row that does not throw SkipRow... But of course that's really the iterator's job, to find the next suitable row for migration. So, I do think next() is the right place, and we'll need to separately address the references to MigrateExecutable from SourcePluginBase.
Just to be clear, I do still think exceptions are better than propagating booleans up the call chain. I've become really tired of
in every D7 migration. prepareRow() will be used less frequently in D8, but it's still a clunky approach.
Meanwhile, looking at SourcePluginBase::prepareRow(), all it's doing now (other than statistics tracking that will be obsoleted by #2443081: Make MigrateExecutable statistics publicly available) is handling trackChanges. This is something that needs to be done after all row preparation is done, but there's nothing to guarantee specific source plugins call parent::prepareRow() last. And typically, at least as used in D7, if you have an intermediate class that you extend, you want to call the intermediate class's prepareRow() first, not last. So, I think the trackChanges handling should be taken out of prepareRow() and be done in next() after the call to prepareRow().
Is it too late to replace calling prepareRow() inline with event listeners? (/me isn't quite sure how serious he is...)
Comment #12
benjy commentedI don't see much benefit in having an exception vs a return boolean if it just to be caught in a parent implementation. However, I don't think it's too late to add an event if we think that's the best approach, it would make more sense to catch an exception that was thrown by an event listener.
Comment #13
mikeryanI'm becoming more enamored of the event approach - let's see how #2535458: Dispatch events at key points during migration goes and if it succeeds look at implementing prepareRow as an event as well.
Comment #14
mikeryanJust a note that adding a prepareRow event should allow contrib to handle idlist filtering entirely on its own, so we'll be able to remove that stuff from core: #2522012: Remove broken idlist handling, replace with more robust exception handling .
Comment #15
mikeryanComment #16
mikeryanKinda feeling like I'm tilting at windmills here - this wouldn't be bad if it were designed in from the start, but I fear it's going to be too disruptive to make the change at this point. Submitting what I have of the patch for testbot, but I didn't get to changing the existing prepareRow() implementations into event listeners.
Comment #17
benjy commentedI don't think we should ever do this. If I'm a source plugin, I have a prepareRow() method that I can overload and just call my parent, I shouldn't have to implement an event listener to do stuff there, that's for others that want to extend/change/react based on what the default source does.
Comment #18
mikeryanSo, I'm going to back off a bit on this and take it in pieces. My immediate goal is to be able to implement the idlist functionality entirely on the frontend where it belongs, and remove the non-functional idlist stuff from core. The first stage of my plan is to do the core-side parts of that in #2522012: Remove broken idlist handling, replace with more robust exception handling (hint: I expect it will involve exceptions in prepareRow()). Once that gets into core, then we can address making prepareRow() an event in core here in this issue.
Comment #19
mgiffordI think that blocker's gone now.
Comment #23
mikeryanComment #24
mikeryanComment #25
mikeryanRealistically, we aren't going to redesign prepareRow() at this stage - let's look at it again in 9.x.
Comment #26
catchIf wanted to add the event, we could do that in 8.3.x (doesn't mean we have to of course, and if it's just hooks vs. events I personally prefer #2237831: Allow module services to specify hooks for that). Bumping down to 8.3.x for that reason, but moving to minor since I don't think this is enabling anything new? The return change seems like the smallest difference here.
Comment #27
OnkelTem commentedSorry I haven't read the whole discussion just having a comment on this and one idea to share (which I've already implemeted):
> SourcePluginBase::prepareRow() should dispatch a PREPARE_ROW event instead of invoking hooks.
Not only that. It should allow to override in child classes. I don't like EVENT system here at all because
1) event handlers are totally unconnected from the migration
2) get called on every sneeze! I.e. there are always multituide of empty calls
3) from (1) -> you have to write if-s to just filter out unnecessary events, not related to your migration.
That said, here is a solution:
1) add checkRow() method to SourcePluginBase:
2) rewrite prepareRow like this:
Now in any Source child we can easily do (example):
OR... we could just return FALSE.
Comment #31
joachim commented> SourcePluginBase::prepareRow() should dispatch a PREPARE_ROW event instead of invoking hooks.
> Catch MigrateSkipRowException in SourcePluginBase instead of relying on a FALSE return.
That's two changes, not one. This should be split into two issues.
> Catch MigrateSkipRowException in SourcePluginBase instead of relying on a FALSE return.
That's a small, easy change, and an improvement to DX, as returning FALSE to skip means that developers have to figure out what to return if you don't want to skip. I've seen lots of migration source plugins that return TRUE here to mean the row is ok, when all they need to do is fall out of the method.
The current behaviour of relying on a FALSE return can be kept for BC.
Comment #32
heddnPerhaps we should re-boot the conversation here on adding an event. See https://github.com/drush-ops/drush/pull/3402 where we are trying to bring things from migrate_run (migrate_tools fork) into drush core. In there, we kinda need an event unless we do really hacky things.
Comment #33
claudiu.cristea@joachim, I split the event constant, event class and event dispatching in #2952291: Dispatch the PREPARE_ROW event. Deprecate prepare_row_alter hooks.
Comment #34
heddnRe-worked the IS to remove the try/catch stuff. If we want to handle that, we can open that in a new issue. This is now only focused on the MigratePrepareRowEvent moving from migrate_plus into core.
Comment #35
heddnComment #36
claudiu.cristea@heddn, in #33 I split the event part in a new issue #2952291: Dispatch the PREPARE_ROW event. Deprecate prepare_row_alter hooks
Comment #37
mikeryanThe scope here now appears identical to #2952291: Dispatch the PREPARE_ROW event. Deprecate prepare_row_alter hooks, so this seems redundant - we might as well close this in favor of the one with the active patch. I'm not particularly inclined to pursue the exception handling changes at this point, but if someone else does they can open a new issue for that.