Job scheduler

  if (!in_array($minimum_period, system_scheduler_get_schedules())) {
    return FALSE;
  }

we never do this. It's documented, and use another minimum_period at your own peril. Other than that I like this. One small tidbit: the idea of encoding the time in the queue name is mine.

subscribing.

Subscribing.

Instead of getting all items you really should use a time limit. It's not a good idea at all to try to dequeue all.

Hmmm. Looks different from the famously useful job_queue module. Alex - take a look there if you haven't already.

There needs to be a really simple way to simply add a function call with arguments to the queue. I don't care which queue or priority, just add it to the pile. No scheduling, recurrence, or anything fancy. Just function name and argument array. It gets called once when it reaches the top of the queue.

while ($item = queue_reserve('scheduler_' . $time, $time)) { while ($item = queue_reserve('scheduler_' . $time, $time) && time() < $when_we_should_finish)

if (!in_array($minimum_period, system_scheduler_get_schedules()))

$schedules = system_scheduler_get_schedules();
$n = count($schedules);
for ($i = 0 ; $i < $n - 2; $i++) if ($schedules[$i] <= $minimum_period && $minimum_period < $schedules[$i + 1]) { break; }

subscribing

FYI: system.queue.inc has changed *a lot* since #10 was rolled... we're in the very final stages of tweaking before RTBC and commit over at #391340: Job queue API. Some things in #10 are definitely not possible any more (not sure they ever were, in fact).

A big example is queue_add() (which is now $queue->createItem($data)) does not return the $item, it just returns a bool if the item was successfully added to the queue or not. I have some misgivings about this aspect of the job queue API, but b/c job queue is pluggable, many possible backends are going to have a very hard time returning the $item in a reasonable way. So, system_scheduler_add() isn't going to be able to return the job queue $item -- at best it could return the $data (what #10 calls $item -- this part of the job queue API used to be a complete mess of confusion -- it's a lot cleaner now).

Also, re: a $lease_time on createItem() -- I don't see that happening. That really makes no sense from the point of view of the job queue API itself. The only timing the queue cares about is the lease for claiming items. The existing queue API has no notion of scheduling tasks (which chx punts to this issue), so there's no concept of delayed claim, submitting tasks "on hold" to be "released" later once they should become available for claim, etc. chx doesn't want to complicate #391340 with stuff like this -- he points me here, but then we can't actually implement that stuff here without API changes over there...

Basically, this issue brings up a lot of the other problems of trying to use the plugable job queue API for any kind of actual scheduling system. :( Job queue API is great for what it's currently designed for: a pluggable fill/drain queue that's optimized for performance and throughput. But, it's pretty useless for a scheduling system, to be honest. I suspect that to build the kind of flexible scheduling system we want for things like the cron tasks and the other things discussed in this issue, we're probably going to need to do one of the following:

• Implement our own queue backend and extend the interface for our needs -- at which point we lose the benefit of the pluggability. Not so clear what we gain from sticking with job queue API in this case, really.
• Implement the scheduler class as its own pluggable interface with a richer API than job queue which will be easier to use for all of this, and harder to implement a backend for if you're using a distributed queue, etc... basically, give up trying to depend on job queue API at all for job scheduler API.

Neither of those sits very well with me. But, I clearly see the value in having a simple fill/drain queue that's not taking on all the complexity of a robust scheduling system. To handle a more complicated thing (scheduling) we need to make more specific assumptions than we need for a generic "pile of tasks" system. Carrying around the assumptions we need for scheduling inside every job queue in core would be a mistake, and make it much less possible to plug in other job queue backends.

Derek, huh? We do claim an item for a time and although it's a bit of abuse you definitely can make it so that the task repeats after the lease expires. That sounds enough for cron.

Got some ideas for this. Interested in the next step :)

@chx: but you can't say "never let this item become available for claim until time X". So, the first time a task gets claimed is ASAP. Thereafter, you can play tricks with leases. It is an interesting approach for (ab)using the leases, but it's definitely a hack. It also depends on promptly resetting items when their lease expires. Currently SystemQueue doesn't do a great job of that, since it uses REQUEST_TIME not time() and only tries to reclaim tasks when hook_cron() runs, not on-demand while trying to claim items. More importantly, without a LOT of churning, I don't see how you can use this system to semi-reliably schedule things at specific times of the day. You have to specify the $lease_time to claim it, but once you've got the item, in general, you'll find that to best fit the schedule, you need to either shorten or extend the lease. job queue API has no methods for manipulating leases.

About the only way to do this given the existing API is to have a *lot* of different queues with most possible options for "how soon from now to start this task?" so that the job scheduler can independently specify "when to first run this" and the period for how often to repeat (if any). E.g. simple example: if it's currently 3pm, and a caller wants to add a task to the schedule that runs every 12 hours at 5(pm|am), you need to first stick it in the "2 hours from now" queue, and when it's done, move it to the "12 hours from now" queue. The granularity of your schedule is directly proportional to N queues you need to create. You might not even need tricks with the lease_time if you did it this way -- you just have to be smart about which queue to put any given item in, and be smart about which queue(s) to try to drain at any given time. However, it requires a lot of queues and a fair bit of complexity. Plus, the accuracy of your schedule will be dependent on the responsiveness of your queue backend -- queues that have higher latency will tend to distort the schedule...

Renamed the issue to better reflect what we want. If we can do a 'do foo every X minutes' thats enough. And if its not precise, oh well.

Renaming back. Actually the cron framework might be a different issue and scheduling at specific points of time might be a valid issue. Noone is forcing us to use the queue API. This issue started as one because i was harassed for a scheduler endlessly in the queue but at the end we might not use the q API but write our own.

Finally downloaded the patch. Going to try and give it an update this weekend.

+

In #131536: Make cron watchdog more granular and informative, it was suggested in #55 to make the cron system pluggable and the issue was marked as "won't fix". Would this be a good issue to discuss that, or should a separate issue be opened, as per #18 of this issue.

Subscribing.

I propose we move this to #1088048: Add ability to schedule actions, since that will be easier to follow and is specific to 8.x. The work here should be moved to that issue summary.