Hi,
I am trying to run a repeating migration and unfortunately the source data does not have an accurate timestamp for when records get changed (why oh why?). Anyway I am keeping track of the relevant data and only want to update records where they have changed. I have taken a similar approach to what the feeds module does in that I am storing an md5 hash of the relevant fields in the destination content type and then returning 'false' at the end of prepareRow() for values which already have identical values in the database, resulting in these rows being skipped. I however don't want to skip these rows but rather just mark them as 'needs_update' in the migrate map, and I can't work out the way to do this.

I hope this is enough information to go on.
Cheers and thanks for a great module.
K

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented
Status: Active » Fixed

If the source key for the row is $row->id, then do:

$this->map->setUpdate(array($row->id));

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

joachim’s picture

joachim CreditAttribution: joachim commented
Version: 7.x-2.4 » 7.x-2.x-dev
Status: Closed (fixed) » Active

Where can that be set?

MigrateSource::next() checks for this before prepareRow() is called.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented
Status: Active » Postponed (maintainer needs more info)

A migration step based on MigrationBase is how I've done this before. In that case, the entities in question didn't have update timestamps, but there was a transaction log, so I implemented a class that ran first which processed the transaction log, deleting anything with a DELETE transaction and setting needs_update to 1 for anything with an UPDATE transaction.

Hope this helps.

joachim’s picture

joachim CreditAttribution: joachim commented

Could the order of things in MigrateSource be changed so this could be set in prepareRow()? It would be much easier that way!

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented

Tough to say, without trying it, whether it would break anything else. Patches welcome...

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Status: Postponed (maintainer needs more info) » Closed (works as designed)
joachim’s picture

joachim CreditAttribution: joachim commented
Status: Closed (works as designed) » Postponed

Could we set this to postponed please?

If mikeryan reckons it might break things, we could perhaps make this change when it's time for a new major version?

This may also turn out to be needed when processing entities with #1883112: Migrate SourceEntity? .

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

Sure - sorry about that! I was just cleaning up some of the queue for Mike yesterday. I'll leave this one alone :)

13rac1’s picture

13rac1 CreditAttribution: 13rac1 commented
Issue summary: View changes

If we are going to leave this open, can someone explain why setting needs_update=1 or using track_changes won't fix OP's issue?

Setting the track_changes option to TRUE will cause each incoming source row to be hashed (after prepareRow) and, if previously imported, compared to the previous value to determine whether the item should be re-imported.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented
Status: Postponed » Closed (works as designed)

track_changes didn't exist at the time of the original issue. I think it obviates the need for any more here...