mapRowBySource executed ridiculous amount of times

Hi there,

I am working on a custom SQL database to Commerce migration that was started by another developer about a year ago. During the process of creating the migration code we updated the migrate module whenever there were significant changes.

For the commerce product migration i am using MigrateSourceSQL as source and MigrateDestinationCommerceProduct as destination. We have seen some performance issues along the way and were able to solve these by adjusting queries, setting proper indexes on source tables and disabling hooks/modules while migrating. For our product migration (100K+) records we are seeing degraded performance as the migration runs.

When running the migration with drush and using the --instrument parameter i can see the following results:

Processed 100 (100 created, 0 updated, 0 failed, 0 ignored) in 34.2 sec (175/min) - done with 'ProductVariant' [completed]
 Timer                                     Cum (sec)  Count  Avg (msec)
 page                                      35.331     1      35330.51
 mapRowBySource                            27.652     89906  0.308
 destination import                        0.581      100    5.812
 entity_save                               0.478      100    4.779
 MigrateSourceSQL performRewind            0.197      1      197.03
 MigrateSourceSQL execute                  0.197      1      196.94
 saveIDMapping                             0.129      100    1.292
 MigrateFieldsEntityHandler->prepare       0.022      100    0.222
 MigrateDestinationEntity->prepareFields   0.019      100    0.192
 MigrateFieldsEntityHandler->complete      0.007      100    0.07
 MigratePathEntityHandler->prepare         0.007      100    0.068
 MigrateDestinationEntity->completeFields  0.004      100    0.041
 MigrateSourceSQL getNextRow               0.003      100    0.029
 ProductVariantMigration prepareRow        0.002      100    0.02

What really stands out here is the mapRowBySource. It is executed once for every previously migrated record and takes about 80% of the total processing time.

I have stripped down the procuct migration to only migrating a title. I have disabled the prepareRow, prepare and complete methods. Unfortunately without any result.

I am not seeing this behaviour in any other migration that i have created so i am reaching a dead end here. Any help would be much appreciated.

Kind regards,
Eric Mulder