Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.This is useful for a large amount of datasource records using Drupal\migrate_drupal\Plugin\migrate\source\ContentEntity
Currently the more datasource records using this plugin; the more memory consumption happens during migrate-import due to eager-loading.
How to reproduce? Have a entity-bundle with lots of fields and that entity-bundle has lots of records (eg.nodes).
$printMem = function () {
$usage = memory_get_usage();
$peak = memory_get_peak_usage();
var_dump([
'memory_get_usage()' => $usage,
'simple usage' => number_format(round($usage / 1024 / 1024)) . ' MB',
'memory_get_peak_usage()' => $peak,
'simple peak' => number_format(round($peak / 1024 / 1024)) . ' MB',
]);
};
$array = [];
for ($i = 0; $i < 1000000; $i++) { $array[] = $i; }
$printMem();
unset($array);
$printMem();
// 50 MB usage, 82 MB peak usage.
// legend: 82 MB is app bootstrap on a drush cli. eg. use (drush php) or (drupal shell) to execute series of php codes.
$node = node_load(8878);
$printMem();
// 52 MB usage, 82 MB peak usage.
use Drupal\migrate\MigrateExecutable;
use Drupal\migrate\MigrateMessage;
$migrationMachineName = 'XXX';
$migration = \Drupal::service('plugin.manager.migration')->createInstance($migrationMachineName);
$migration->getIdMap()->prepareUpdate();
$executable = new MigrateExecutable($migration, new MigrateMessage());
$executable->import();
$printMem();
// 439 MB usage, 439 MB peak usage.
| Comment | File | Size | Author |
|---|---|---|---|
| #24 | interdiff_21-24.txt | 761 bytes | pooja saraah |
| #24 | 3158436-24.patch | 4.13 KB | pooja saraah |
| #23 | 3158436-nr-bot.txt | 1.62 KB | needs-review-queue-bot |
| #22 | interdiff_20-21.txt | 1.07 KB | shivam-kumar |
| #21 | 3158436-21.patch | 4.26 KB | shivam-kumar |
Issue fork drupal-3158436
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes
Comments
Comment #2
joachim commentedI've run into this problem with a ContentEntity migration that has a large number of records to migrate.
With about 2 million source entities, my server ran out of memory with 512MB. This happens even if I try to migrate a single entity with the --limit option to the 'drush mim' command:
> PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 167772160 bytes) in docroot/core/lib/Drupal/Core/Database/Statement.php on line 112
[warning] Drush command terminated abnormally.
I suspect the problem is here:
Before the migration even begins, the iterator is initialised, and here, we load EVERY entity ID for the WHOLE source.
An array of 2 million entity IDs is actually going to be pretty big! Going by what it says on https://stackoverflow.com/questions/36709580/what-is-the-actual-memory-c..., it's probably going to be something like 400MB, which would explain why my server ran out of memory!
I'm changing this to a bug, as without this, the ContentEntity migration source doesn't scale to large datasets.
Comment #3
joachim commentedI've added a batch to the query in yieldEntities(), and I'm trying a large migration with a debug statement that shows memory_get_peak_usage(). The result shows that memory keeps creeping up:
Processing about 1000 source entities adds about 57 MB to the memory usage, so 20K entities is going to be 1GB of memory.
I'm fixing this by forcibly clearing the entity memory cache every batch.
MR coming!
Comment #4
joachim commentedWith the clearing of the entity memory cache, memory is creeping up, but much more slowly:
About 200k memory per 1000 source entities, looking at the figures that come later on. So 5k entities will add 1 MB, and and to use 1GB of memory will take 1M source rows.
Comment #7
joachim commentedOpened merge requests for 9.2 and 8.9 (the changes are the same in both branches).
Comment #8
quietone commentedThis needs a test.
Comment #12
larowlanComment #14
_pratik_Comment #15
_pratik_Attached the rerolled patch for 10.1.x.
Comment #16
nivethasubramaniyan commentedFixing CCF in #15
Comment #17
sandeepsingh199 commentedTried to fix ERROR of #15 and #16.
Comment #18
sandeepsingh199 commentedattaching interdiff
Comment #19
medha kumariRerolled the patch #17 in Drupal 10.1.x.
Comment #20
_utsavsharma commentedRerolled the patch for 10.1.x.
Please review.
Comment #21
shivam-kumar commentedErrors in #20 have been fixed in this Patch and rerolled for 10.1.x.
Comment #22
shivam-kumar commentedAdded interdiff for #21.
Comment #23
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #24
pooja saraah commentedFixed failed commands on #21
Attached patch against Drupal 10.1.x
Comment #25
joachim commentedFailure looks unrelated.
Running this patch on 10.0 -- thanks for the reroll!
Comment #26
smustgrave commentedSeems the retest aborted also. I don't think there is a build problem but could be wrong.
Am moving to NW for the tests.
Comment #27
berdirThe while (TRUE) here seems wrong, you don't need a while with yield, otherwise it will run forever, that's why all the CI runs time out.
Also, we can add a loadMultiple() because loading 50 nodes at once will result in ~50x fewer queries.
Comment #28
joachim commentedWe need to end the loop when $ids is empty, but we can only determine what $ids is once we've run a query.
It's not something we can put inside the condition for the while().
Hence why I went for the while (TRUE) { break; } construction.
What do you suggest instead?
BTW this issue started with a MR, please could we stick to that and not post patches?
Comment #29
berdirYou are right. This would be much nicer as a
do { ... } while (empty($ids)though.That said, after reading it again, I see the problem. it only works if you do have a batch size, because without a batch size, it will not add a range and then you will run the same query over and over again and $ids will never be empty. You could do a count($ids) < $batch_size || !$batch_size or so.
The thing with batch size is that it's quite tricky. Rerunning the query can give you different results, it is for example fundamentally incompatible with the map table join that you're also working on. The map table join will no longer return the results that already match and then the batch would also skip non-processed results, causing you to skip $batch_size * $n on every query execution. Additionally, the query currently has no sort, so some databases will not guarantee a stable sort order, again risking to only process a random subset, so you'd need to add a sort. That's why \Drupal\migrate\Plugin\migrate\source\SqlBase::initializeIterator() is so complicated, it's handling all those different features and can only use some combinations of them.
So you have to pick between using the map join to speed up reruns (and then you can do multiple batches of 100k/1M or so entities) and a batch size anyway.
IMHO, using loadMultiple() and the memory reset will have a much bigger impact on memory usage, although a large enough data set will eventually make you run into a hard limit as well, but most people will hit the limit much earlier due to loaded entities. Based on a blackfire run against a data set of 420k source entities, Drupal\Core\Entity\Query\Sql\Query::result() uses about 250MB for that. I did some tests with range(0,1000000) which uses only 33MB, but the query gives us strings which clearly use up much more space.
What do you think about focusing on that first, possibly in a new issue. I'm confident that we can get that into core much more easily as there shouldn't be any negative side effects/possible bugs unlike batch. Maybe we need a test with more entities than the chunk size, but that's about it. We could maybe use
Settings::get('entity_update_batch_size', 50), or a different setting name, so we don't have to create 50 entities in a test.For those tests, I compared using array_chunk(), array_splice() and array_slice(). array_chunk() more than doubles memory usage as it creates many smaller arrays, array_splice() is extremely slow for this use case (get the first N entries from the array) probably because it reorders the whole array on every call, array_slice() with an incrementing position looks like a clean winner:
Comment #30
mathilde_dumond commentedI created a related issue and uploaded a patch that runs the query in one time, but loads the entities by batches, and clears the memory regularly. It may not be enough for all situations, but already helps with memory and performance #3354201: Use loadMultiple() and reset memory cache in ContentEntity source
Comment #31
joachim commentedIt's been a while since I've worked on all this, and my current migration project is a low-budget affair so not much more than rerolling patches. So I don't remember all the details.
> IMHO, using loadMultiple() and the memory reset will have a much bigger impact on memory usage, although a large enough data set will eventually make you run into a hard limit as well, but most people will hit the limit much earlier due to loaded entities
I hit the memory limit from just getting an array of entity IDs from the query on a migration that was 6M source entities. That sounds like a lot, but it was just a customer portal website. It's not an outlandish use case.
> Rerunning the query can give you different results, it is for example fundamentally incompatible with the map table join that you're also working on. The map table join will no longer return the results that already match and then the batch would also skip non-processed results, causing you to skip $batch_size * $n on every query execution.
Took me a while to get my head round that, but yes, I see what you mean.
What's strange is that I definitely used batch_size AND map join (with the patch from #3188914: ContentEntity migration source doesn't consider the migration map) and I don't remember having problems with it. Maybe there was some query caching going on in mySQL? Or maybe my map join patch wasn't working properly?
At any rate, that's something that could be figured out - don't advance the LIMIT part of the query if the source uses a map join.
The map join is important too, as without it, resuming a migration after a crash takes a horrendous amount of time as it checks everything it's already migrated.
> Additionally, the query currently has no sort, so some databases will not guarantee a stable sort order, again risking to only process a random subset, so you'd need to add a sort.
#2845863: Migrate SQL source query isn't ordered :)
> What do you think about focusing on that first, possibly in a new issue. I'm confident that we can get that into core much more easily as there shouldn't be any negative side effects/possible bugs unlike batch.
You mean doing a separate issue which just handles clearing the entity storage static cache, and postponing this issue to be worked on after the new one? Yes, makes sense to split this up into simpler issues. Looks like @mathilde_dumond has already created it -- thanks! :)
Comment #32
joachim commentedIn which, this is postponed on #3354201: Use loadMultiple() and reset memory cache in ContentEntity source.
Comment #34
benjifisherIn #3498915: Move content_entity source plugin to migrate module, we moved the
content_entitysource plugin to themigratemodule. I am adding that as a related issue.