There is currently a 'feeds_process_limit' setting that allows you to restrict the number of items imported by a single cron run. This has a couple of problems:
- It is a site-wide setting, but entities on different feeds may take vastly different times to import, and therefore need different settings.
- It doesn't take account the performance difference when items are skipped because they haven't changed since they were last imported.
- At best, it should be set to a number of items that can be imported in slightly less than php's max_execution_time, to import as many items as possible without the script being terminated.

It would be better to have a 'feeds_process_timelimit' setting. This would probably need changes to each importer for it to be respected. I'd suggest a global time limit, that is equally divided between all importers that are waiting to import any items. e.g. if you set the time limit to 20 seconds and have three importers configured:

import_a: Fully imported
import_b: 9,999 of 10,000 items imported
import_c: 0 of 10,000 items imported

then it would give 10 seconds to import_b, and 10 seconds to import_c. Next time cron ran you'd expect import_b to have completed, so import_c would get 20 seconds.

The new setting would need to be in addition to the feeds_process_limit setting, as it could only apply to the importer stage. If a feed takes too long to process, that would need to be controlled with the old setting.

Comments

MegaChriz’s picture

Sounds like a nice idea, but maybe this should be handled in #1363088: Feeds batch process not fully implemented - needed for large imports? Cross-referencing for now.

ianthomas_uk’s picture

Issue summary: View changes