Problem/Motivation
If one tries to execute a migration import where there are higher number of migration plugin instances, then it is easy to run out of memory. This mostly happens because:
- When a migration plugin instance object's requirements are checked, then the source and destination plugins of the migration are instantiated too. These consume a LOT of PHP memory, since these not only contain the injected services (entity type manager, field manager, field migration manager etc), but often also their migration "host" object.
- When a certain migration's required migrations are checked, then the migration plugin manager always builds the optimal order even of the required migrations - this is unnecessary too. We only need the optimal migration execution order when we start the import (or the rollback) process.
Steps to reproduce
Create a migration environment with about 5000 migration plugin instances -> you will need at least 4Gb PHP memory just for being able to execute drush ms.
Proposed resolution
Outsource requirements checking into lightweight helper functions, which are performing the necessary checks on a cloned migration object - which then gets unset before the method gets finished.
Remaining tasks
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | smc-multithread_improve_memory_management-3307033-2.patch | 44.28 KB | huzooka |
Comments
Comment #2
huzookaMemory requirement (peak memory usage) has dropped to ~180Mb with the same amount of migration derivatives!
Comment #4
huzooka