Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Problem
I was attempting to go through a limited migrate-import to debug it. The migration shows over 800 records in it. The first step works fine:
$ drush mi menumigration --limit=100
Processed 100 items (100 created, 0 updated, 0 failed, 0 ignored) - done with 'menumigration' Trying to run the migration again with a limit only imports a single record:
$ drush mi menumigration --limit=100
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'menumigration'This occurs whether or not the limit is changed in subsequent runs. Running the migrate-import with no limit will complete the migration correctly. I originally found the issue on 8.x-2.0-beta1 but it is still present on 8.x-2.x-dev.
Proposed resolution
Issue is that the limit is not factoring in the records already processed. I'll be attaching a patch shortly.
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | drush-migrate-import-multiple-runs-2730081-2.patch | 686 bytes | dhansen |
Comments
Comment #2
dhansen CreditAttribution: dhansen as a volunteer commentedComment #3
draenen CreditAttribution: draenen at Monarch Digital commentedPatch applies and works for me.
Comment #4
mikeryanActually, that's not ideal. The basic problem is that prepareRow gets called for already-imported items, so the counter is not reflecting what's actually been "processed". And I'm not sure in all cases prepareRow will always be called for all previously-imported items first. I'm thinking the itemLimit code should be using getProcessedCount() rather than $this->counter.
Comment #5
dhansen CreditAttribution: dhansen as a volunteer commented@mikeryan I'm a little unclear about your comment as the patch already adds the use of getProcessedCount(). Is there somewhere else in the code this needs to be altered?
Comment #6
dman CreditAttribution: dman at Sparks Interactive commentedI've been getting this and it's been killing me for three days - I blamed all the weird and wonderful things I've been doing to my own imports. I could have sworn it used to work as advertised last month when I did my proof-of-concepts for my current job - but I've been blaming myself for messing something major up, and have been going down to db dump diffs and step-debugging to try and find out what I did wrong.
It's a relief (and also somewhat exhausting) to find out *it's not just me*! Auuuugh!
(migrate_tools 8.2.0-beta-1)
Comment #7
dman CreditAttribution: dman at Sparks Interactive commentedAdditionally, the observation that "prepareRow gets called for already-imported items" goes a loooong way towards explaining my other issue that lost me most of last week - it seemed that the process was incrementally slowing down massively as my runs progressed ... because I thought that I could avoid scaling issues by running repeated batches of nodes (out of 60,000) - processing 1,000 at a time. But the slowdown looked like a memory leak or something. And I spent days trying to isolate *that*.
Patch appears to solve my current problem today, thanks. I'm lost in trying to imagine the "real" solution though. I had imagined there was a high-water-mark or an iterator pointer remembered somewhere, but my investigations so far have not been able to find it.
Comment #9
mikeryanFixed by making use of getProcessedCount() in onPrepareRow(), thanks!