Problem/Motivation
During the migration the some data can be migrated as failed. I know that one of the migrate plugin throws an exception with a detailed message. But after the migration is finished only summary is presented (X imported, Y failed). The messages are
empty. After debugging I found out that messages are cleared every time in Drupal\migrate\Plugin\migrate\source\SourcePluginBase::next() file core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php:
// Clear any previous messages for this row before potentially adding
// new ones.
369: if (!empty($this->currentSourceIds)) {
370: $this->idMap->delete($this->currentSourceIds, TRUE);
371: }
So, every time as the new row will be migrated the next() method will iterate over previous rows and it will clear their messages.
Proposed resolution
I think that the message should be cleared before the row is scheduled to import.
User interface changes
No visible changes.
API changes
None.
Data model changes
None.
| Comment | File | Size | Author |
|---|---|---|---|
| #31 | drupal-migrate-no-messages-after-migration-3036368-31.patch | 4.39 KB | martin_klima |
Issue fork drupal-3036368
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
skoro commentedComment #5
khiminrm commentedI've faced with the same issue. Proposed solution works for me. Created the patch.
Thanks @skoro !
Comment #6
quietone commentedAdded a test. No interdiff because the difference is just the new test.
The one thing that needs to be considered here is that previously the message row was delete before prepareRow() was executed and it is not done after. What problems might this cause?
Say, that on the first run the row caused an message to be written to the message table. Then a prepareRow() method/hook/event was added that caused that row to be skipped and the migration was run again. In this case, the message would still be in the map when it should not be. It is perfectly reasonable to expect the message row to be removed.
edit: remove reference to deleting the map row.
Comment #9
wim leersI wrote at #2859314-51: Highwater condition with unjoined maps skips unprocessed and NEEDS_UPDATE rows how you could test this with a unit test.
Comment #11
kiseleva.t commentedIf I execute migration 2 times, there will be duplicated messages.
I've moved the deletion under condition that it's current row and it works fine, each message presented only once.
Comment #15
guypaddock commentedThe test failure from
Drupal\Tests\field\Kernel\Migrate\d6\MigrateFieldTestwould seem to indicate that messages are still getting wiped out when they shouldn't be. Perhaps this is the wrong spot for messages to be cleared in the first place -- why does iterating over the data set result in mutation of messages?Comment #16
guypaddock commentedTaking a crack at this now.
Comment #18
guypaddock commentedI've tried to fix this up. My understanding of the original code is that, for each batch of records being processed, the iterator is used to seek through the source until the first row that has not been processed is found -- clearing messages for the prior rows along the way, which is not desired. The solution in #11 was to clear the messages once we know the row is being processed/needs update, BUT this is happening after the source plugin has been invoked, which is also not ideal because we can lose messages emitted by the source plug-in.
Ideally, at the start of a migration execution, we'd make a pass over the whole result set and clear messages for all records that already have a source ID in the map, but I could not come up with a way to do this without a BC break on the
MigrateSourceInterface. Alternatively, maybe there's a way to signal to the iterator that we are seeking and don't want to clear messages. Either way, it feels like we'd need a BC break.Instead, what I ended up doing was to still clear messages before invoking the source plugin (as Core was doing before this issue) but re-using the same conditional logic as is used after the process plugin so that we only clear messages for rows we know need processing. I am not 100% sure if this covers all the cases we care about, or even if some of the flags being set for those conditionals would be appropriate before invoking the source plugin, but it appears to pass both the test that was failing previously as well as the test introduced by this patch.
Comment #19
guypaddock commentedThis was the test failure:
It looks like this deprecation warning is being triggered by the test runner trying to print out an error. The error is actually:
But... that too is a red herring. That error is being thrown in
\PHPUnit\Util\PHP\AbstractPhpProcess::processChildResult()in the test runner when it's trying to decode a test failure caused by a PHP notice (it's trying to unserialize and wakeup all of the database connections referenced by the Migrate test, which includes a database connection calledmigrate, which the test runner doesn't have.)The actual error was:
I'm working on that now.
Comment #20
guypaddock commentedFixed!
Comment #22
mxr576Comment #23
mxr576Comment #24
smustgrave commentedThis issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.
Adding specific steps would help to see the issue.
Hiding the files as the fix is in the MR (which needs updated too)
Tagging for subsystem review to see if they have any thought about it too.
Comment #25
codebymikey commentedAttached a copy of the latest static patch for reference purposes, and it seems to be an improvement over the existing behaviour on 10.0.x, when used with the
migmag_processmodule.Comment #26
martin_klimaThank you for the patch, but it does not work well if "Tracking changes" is enabled. There is a logic problem when the record was previously migrated and is stored in idMap.
Before deleting a record in the migration messages table, it tests if the record has been changed ($this->rowChanged($row)). However, at this point, we cannot evaluate if the record has been changed because there is no 'original_hash' in the 'idMap'.
The original hash is stored a few lines later, at the end of the prepareRow() method, where the rehash() method is called. The current logic always evaluates an existing record in the migration map as changed, because there is no original hash. See \Drupal\migrate\Row::changed and \Drupal\migrate\Row::rehash.
Thus, it deletes the migration messages for such a record, even if it is later skipped, because on the second test with the
$this->rowChanged($row)method, the original hash is already present and the record is correctly evaluated as unchanged. As a result, the record is ignored as unchanged, but all previous migration messages for the record are deleted.We can solve the problem at least in 2 ways:
1) Remove
$this->rowChanged($row)from the 1st condition.2) Modify \Drupal\migrate\Row::changed so that it only evaluates the record as changed if there is the original hash in idMap.
Both ways work, but I'm not sure which approach is better and safer.
Steps to reproduce the problem:
1) Use a migration with trackChanges=TRUE
2) Migrate some records which generate some migration messages (e.g.
drush mim migration_name --limit=10).3) Check migration messages in the DB table or via
drush mmsg migration_name4) Repeat step 2
5) Repeat step 3. You can see the previous messages are gone.
Comment #27
martin_klimaAttached a copy of the latest patch for reference purposes.
Comment #28
codebymikey commentedRe #26, this is currently untested "pseudo-code", but wouldn't the following work a bit better rather than not checking if the row has been changed and always keeping the previous messages?
We could either use this approach or update
Row::changed()so that it recalculates the hash iforiginal_hashis NULL, mitigating this issue.Might be worth applying this approach and seeing if it addresses your use case. Not sure if rehashing can cause other issues.
Comment #29
martin_klimaRe #28
My approach is not ideal because it prefers to keep messages in case they can be deleted (better to keep than delete).
But your suggestion will not work either.
The problem cannot be easily solved because the steps must be executed in a certain order, but this order cannot be defined due to constraints.
1) Deleting messages cannot be before $row->changed() because we don't know whether the record will be processed.
2) We cannot check $row->changed() before the end of prepareRow(), because source data is not stable yet, e.g. hook_migrate_prepare_row can modify the source data.
3) We cannot delete messages after prepareRow(), because prepareRow (and hooks and child prepareRow methods) can save some messages.
Where to delete messages if we know that:
- Deleting messages cannot be before $row->changed()
- Deleting messages cannot be after $row->prepareRow()
- $row->changed() cannot be before row->prepareRow()
This logic is unresolvable.
The only idea I have is:
- Backup messages for the current row
- Delete messages for the current row
- Test if the row will be processed
- If the row will not be processed ($this->currentRow == NULL), restore messages back.
Comment #30
martin_klimaIn the last commit, I reverted the 1st condition, so migration messages for a current row are deleted in the original way. But they are stored in the backup variable before deleting. If the row will not be processed (was migrated before and does not need update and is not above the highwater mark and was not changed), all deleted messages are saved back from the backup.
In this way, we don't need to solve the logic problem with impossible checking if the row has changed before processing prepareRow(). For details, see #29.
Tested with $trackChanges = TRUE;
Comment #31
martin_klimaFixed undefined variable.
Comment #32
mikelutzThis needs to have the merge request updated to 11.x, or a new MR created against 11.x
My only concern with the approach is that the prepare row method and hooks can also write messages to the table, and if they aren't explicitly skipping the row by returning false then any messages created by the row preparing will start accumulating. Maybe we should check whether any messages exist for the row before we restore instead of just checking if the row will be skipped due to track changes/highwater/needs update etc.
I was also a little worried about performance, I ran some tests on a 40k row migration without the map table joined. The migration was completed, so no rows were actually processed. I inserted a message for each row and ran a few times with and without the patch. I was seeing ~15seconds or 6% additional runtime on average with the extra database trips per row. It's not insignificant, but it could be worse, it extrapolated to an extra 6ish minutes on a million row run that would have taken an hour and a half already, so maybe it's worth providing a way to opt out? I'd approve the MR without an opt-out, but I'll let others weigh in.
Comment #33
joekersThanks @martin_klima for the info and patch - I'm in the same situation as you with tracking changes.
The patch is working well for me but the performance is a bit concerning, though it's more important that I have the messages kept as I run a set of migrations daily and need to know when there are issues.
Comment #37
benjifisherI ran into this issue while testing Import CSV Recipe.
I created a new branch, rebasing the feature branch on 11.x. I did some manual testing, and it looks good so far.
If I have time, I will review the code and/or make some updates myself. For now, back to NR.
Comment #38
benjifisherThere are some failing tests.
I fixed one: now that we call
getMessages()inSourcePluginBase::next(), the mockIdMapobject has to be told to expect that.The other failures are more serious. The functional upgrade tests now log almost twice as many messages as expected in
testUpgradeAndIncremental(). That test function migrates data from the D6 or D7 database; adds some content; migrates again (acknowledging the warning that there could be data loss). In the 11.x branch, the second migration logs only one message; with the feature branch, the second migration logs as many messages as the first migration.I do not see why the second migration logs more messages using the feature branch.
Comment #39
benjifisherMy last commit implements the original proposal for this issue. Instead of saving and restoring messages, it deletes them a little later in the process.
I still do not understand why the previous version made the upgrade tests fail, but this approach seems to fix that.
Another advantage: this approach should avoid the performance problem raised in Comment #32.
Unfortunately, this approach runs into the other problem mentioned in Comment #32 (or maybe the mirror image of that problem): it deletes any messages added in
prepareRow()or in the hooks that method invokes. This leads to a couple of test failures.Comment #40
benjifisher