Problem/Motivation
There was a bug report related to problems with the row hash when using track_changes and migrating using the --update option. Tested in #5 and was not able to reproduce.
Proposed resolution
Add assertions to TrackChangesTest
Remaining tasks
Decide if new test coverage is worth committing.
User interface changes
None
API changes
None
Data model changes
None
Release notes snippet
None
Original Report Below
Problem/Motivation
During a migration process, the Row object contains $source property containing source data. The hash is used for checking if the source data have been changed since the last migration. The hash is calculated from the whole $source property.
When a record is imported for the first time, the $source property array keys migrate_map_sourceid1 and migrate_map_source_row_status are NULL.
When the same record is imported again (e.g. as migrate import with --update parameter), then the keys migrate_map_sourceid1 and migrate_map_source_row_status are not NULL anymore and these changes cause the calculated hash to be different. It means the record is considered as changed, but the source data have not.
Steps to reproduce
Set $this->trackChanges = TRUE, migrate one record, check the hash in the migrate_map.
Migrate the same record again (as --update or --idlist) and check the hash again.
The hash has been changed. It shouldn't, because the data are still the same.
Proposed resolution
Use the same logic as D7 Migrate module and exclude migrate_map_* data from the hash calculation.
Remaining tasks
Would be nice to know what is adding 'migrate_map_' properties to the source
| Comment | File | Size | Author |
|---|---|---|---|
| #11 | 3240873-11.patch | 2.58 KB | quietone |
| #11 | interdiff-9-11.txt | 1.17 KB | quietone |
| #9 | 3240873-9.patch | 2.51 KB | quietone |
| #9 | interdiff-7-9.txt | 1.73 KB | quietone |
| #7 | 3240873-7.patch | 1.84 KB | quietone |
Comments
Comment #2
martin_klimaComment #3
martin_klimaFix patch: Add missing row.
Comment #5
quietone commentedWhat is adding migrate_map properties to the source?
I tested this on Drupal 9.4.x, standard install and was not able to reproduce the problem.
I used the Drupal 7 test fixture as the source and the d7_user_role migration, with
track_changes: trueadded. I randrush mim d7_user_role. I then used the debugger and checked the hash in Row::rehash() before and aftermim --update d7_user_role.Maybe not needed but I added assertions in TrackChangesTest to confirm the hash does not change when the source does not change.
Comment #6
quietone commentedFix spelling error.
Comment #7
quietone commentedRemove an unneccesary change.
Comment #8
danflanagan8That's a great question. This can't be a core source plugin right?
As for the new assertions, it looks like there's some kind of copy-paste typo below:
I'm thinking this was intended to be more like this:
I also have a question. The IS and comments refer to using the
--updateoption when runningdrush mim, but the test obviously doesn't rely on drush. Do we need to add$id_map->prepareUpdate();somewhere(s) in the test?FWIW, the new assertion still pass if I throw add
$id_map->prepareUpdate();before the re-migration.I'll set this to NW to fix that copy/paste typo, but then it should be good. Thanks!
Comment #9
quietone commented@danflanagan8, thanks for the review. Good point about using prepareUpdate.
#8. It turns out the both 'Item 2' and 'Item 4' are modified so two of the hashes change. At the end of the test I added the prepareUpdate() and retested.
Comment #10
danflanagan8If we add the re-import after calling
prepareUpdatethe assertions would be more useful if they were allassertEquals. TheassertNotEqualsassertions are kind of funny here.Maybe this last foreach could store the has in a variable called
$newer_hashsuch that each of the assertions would look like$this->assertEquals($new_hash[#], $newer_hash[#]);.Comment #11
quietone commentedWell, that was a real oops. I did mean to execute the migration again after the prepareUpdate.
#10. Done. I agree, it reads better.
Comment #12
danflanagan8Double oops: I hadn't even noticed the migration didn't get run again after preparing for update.
The test certainly looks good now. So what do we do here? You've proven the reported bug is not a problem in core. Do we change this to a task for adding coverage related to the hash?
I think it's reasonable coverage to add. The only place I can find explicit coverage related to hashing is in
Drupal\Tests\migrate\Unit\RowTest. The new coverage is testing for unexpected interaction betweentrackChangesandprepareUpdate, which seems ok to add.I'm going to update the IS and category and RTBC.
Comment #13
quietone commentedHow about changing the title to what this is actually doing.
Comment #14
alexpottCommitted and pushed be8ec278470 to 10.0.x and 83394db7aa6 to 9.4.x and 7bd674b5249 to 9.3.x. Thanks!
Backported to 9.3.x as additional test coverage is always welcome.