Hosting cron # of sites per run mismatch between UI and reality

I'm not sure it is related to Barracuda, as main Hostmaster instance is running vanilla Aegir system, including cron settings.

Note that Aegir will never run cron for *all* sites every minute if the number of sites is higher than Aegir default threshold (calculated internally). This has nothing to BOA anyway, I'm afraid, unless you believe that this mod affects anything here: http://drupalcode.org/sandbox/omega8cc/1074912.git/blob/HEAD:/modules/ho...

Please try to debug this and move the issue to Hostmaster queue maybe.

I'm, not sure this is still relevant ... but any new development will have to be on 7.x-3.x or 4.x
--
This is a templated response, please re-open or comment if you think it's in error.

So the stuff around 'batch' queues like the cron queue is really weird!

The maths does work out but the labels being used are super confusing.

Suppose I have 100 sites and I want cron to run on them once a day, at the moment, the maths will mean that:

1. Every 4 hours the cron queue will be processed.
2. Each time the queue is processed 17 cron invocations will run in serial.

This means that over the 24 hours 102 cron invocations will indeed happen.
However, the cron runs will peak at the 4 hour points, and nothing will happen in between, unless it does actually take 4 hours to run those 17 crons.

The stuff about 'threads' is a bit misleading I think. There's no real sense in which the cron queue is ever going to spin up 6 threads and execute cron on them in reality.

I reckon we could do a better job of spreading the load over the period selected. Not sure of the exact maths required though :)
(I have a Mathematics degree)

See this image:

This is basically showing the the Aegir cron implementation on this server runs cron sort at 2 hour-ish intervals with relatively big peaks, rather than spreading it out into the massive gaps in between.

We have replaced the stock cron module with ported to D7 hosting_advanced_cron. It is currently tweaked to run with BOA and should be improved to make it work with vanilla Aegir, but maybe it is worth to look at?

It looks like that module still relies on the code in Aegir's dispatcher to start cron at specified intervals, so it would still need addressing in hosting.

Yes, correct. We don't experience these spikes only because people usually define custom cron schedule per site, plus, the module doesn't allow to run cron for too many sites at once, while globally we set hosting_queue_cron_frequency 1 and hosting_cron_default_interval 86400

It is rather a side note, because I realize that such change would require Aegir 4.x, while the fix for the current logic can be implemented in 3.x

So maybe we'll get rid of all this anyway once we have #422962: Add Hosting Tasks Schedule module squared away, but here's a very initial attempt at implementing a (badly named) 'parallel' queue that spreads the load evenly over the entire time span as much as possible.

Going back to my earlier example:

Suppose I have 100 sites and I want cron to run on them once a day, at the moment, the maths will mean that:

1. Every 4 hours the cron queue will be processed.
2. Each time the queue is processed 17 cron invocations will run in serial.

But now:

1. Every 14.4 minutes the cron queue will be processed.
2. Each time the queue is processed 1 cron invocation will run in serial.

As you mention, "parallel" doesn't seem to be a good name for this. Maybe "spread", "diffuse", or one of it's synonyms? "Equalized"? "Flattened"?

There are a couple magic numbers in there. Can we convert those to descriptive constants? 86400 is obviously the number of seconds in a day, but 120 seems arbitrary. Maybe it's just me, but I'm having some trouble actually parsing what it's supposed to be doing there. Could we add some docs?

Also, this appears to be a change in the default behaviour. If we re-interpret this as a bug report, I think we can justify that change. Otherwise, we may have to hold off until Aegir 4.x (which would be a shame).

Spread, I like it.

Here's a patch that:

• Adds a bunch more documentation for all queue types.
• Introduces a new set of defines for the queue types.
• Doesn't change the type of the cron queue.
• Introduces a couple of new hooks for playing with the queues.
• Means that people like me can implement a quick hook to change the queue type from batch to spread.

(We can then change the cron queue type to be spread in 4.x)

Sorry, bad patch.

There appears to still be one stray reference to "parallel": * @see HOSTING_QUEUE_TYPE_PARALLEL. Otherwise, the code looks great! I like the new hooks. It'd be pretty trivial to add a little "hosting_cron_spread" module that invokes that hook, to allow users to opt-in to the new behaviour right away.

I'm leaving in "Needs review" for the time-being, since I haven't had time to actually test this. +1

Oh, good spot, here's that change.

Gah, missed another one.

It looks good, and as it does not changed the current default ...

Committed.

PS: The 'type from batch to spread' change might also make for a nice example in the hosting.api.php file.