I have a test setup with these queues defined in hook_cron_queue_info:
- queue_1, time = 2
- queue_2, time = 2
- queue_3, time unset
I run:
drush meq 20
What I expect are the following run times:
- queue_1: 2s
- queue_2: 2s
- queue_3: 16s (remaining time to reach 20s in total as described in command's help text)
What I get is:
- queue_1: 2s
- queue_2: 2s
- queue_3: 20s
As of now, the command's 'time' argument acts as a queue-level default queue processing time if there isn't any defined for the queue in hook_cron_queue_info. What I would expect is meq command 'time' argument to control total run time for the command.
After the above test, I did another one: I just changed queue_3's 'time' from unset to 30 and ran drush meq 20
again. What happened is:
- queue_1: 2s
- queue_2: 2s
- queue_3: 30s
With the above run, we've exceeded the 20s desired total run time.
- Do 'time' command argument refers to total run time or queue-level run time if none set?
- If it is meant to be total run time, could it be exceeded in any circumstance?
- Could there be a switch to allow exceeding (or not) it? Which should be the default?
In any case, I think both time parameters would be desirable to have, so I would happily submit a patch. But first I would like to sort out which improvements are needed clarify existing code's functionality.
Having a total run time (which could not be exceeded) kind of setting would be useful in my setup for running meq in Google Computing Engine's preemptible instances.
Also, a "queue-level run time in case none is defined" kind of parameter would provide extra tweaking for complex scenarios. For example, giving less processing time by default to queues with unset time (ie. less important) or providing finer-grained queue processing time slices.
Comments
Comment #2
wuinfo - Bill Wu CreditAttribution: wuinfo - Bill Wu for Wulei Info Technology Solutions commentedPut the time settings in the queue functions, where the implementation of hook_cron_queue_info is.
queue_1: 2s
queue_2: 2s
queue_3: 30s
"drush meq 20" is not able to control the individual queue execution time. For the overall time, it also depends on the setting in PHP.ini file. It is likely to exceed 20 seconds in command "drush meq 20".
"if the timeout is the default 30 seconds, and 25 seconds into script execution a call such as set_time_limit(20) is made, the script will run for a total of 45 seconds before timing out"
http://php.net/set_time_limit
https://api.drupal.org/api/drupal/includes%21common.inc/function/drupal_...
Comment #3
wuinfo - Bill Wu CreditAttribution: wuinfo - Bill Wu for Wulei Info Technology Solutions commentedIt is not going to perfect to control the individual queue execution time. Like firing the rocket, what we can do is to manage the period that will fire the rocket, but not has control of when the rocket will land at the destination.
Comment #4
amonteroI'm aware that the execution time is not under the module's control, since each worker function will take different times for each call, neither I want to do such fine-grained control. I'm OK with the times being coarse, or at least an approximation.
Anyway I just want to make sure which purpose is meq's time parameter, first to improve the command help documentation. Also, clear clarifications for the above questions will help me understand the code better to avoid breaking any contract with the user when adding new features.
After that, I will retest everything again to see if there is actually any problem with the results I get. I will get some time soon to look at it again and would like to have it clear before going with it.