dispatcher locking

Another fun thing that can happen is that the multiple tasks launched by the dispatcher all decide to apachectl graceful apache, which will take non-negligible time, especially under load and/or with a lot of vhosts. We need a better way to reload apache...

A LOCK TABLES ... READ should probably be aquired on the queue so that only one dispatcher runs at a time. This seems like the most elegant solution. Of course, if we still fire tasks in parallel, then we will probably still have the problem of task buildup, so I would suggest that the dispatcher (or at least the task dispatcher should *stop* running in parallel).

Even if the dispatcher is light, it still has to bootstrap Drupal to get the SQL credentials, which does mean significant load.

Furthermore, if it just fires and forgets tasks, it can't keep track of which tasks started and finished, so tasks could just build up.

There are two degenerative cases:

1. the dispatcher overload: the first dispatcher has not finished loading and a new dispatcher starts, which also bootstraps drupal and those cronjob stack up to takes up all resources
2. the task (or cronjob) overload: the dispatcher runs normally, but starts too many tasks at a time, which stack up and take up all the resources.

Of course case 2 can degenarate into case 1 for maximum overload.

Case 1 may be solved by a lock on the table, but I'm not sure: to get there it needs SQL, and to get to SQL, it needs to boostrap Drupal, which is the main issue here. Maybe a filesystem-level lock would be better here instead.

Case 2 will *not* be solved by a lock on the table. We *could* examine the state of the queue to see how many tasks are currently running and fire only the tasks we're allowed to run. So say there are 3 tasks still running and we're allowed to run 5, we would fire up the two next tasks.

I would rather, however, have a simple, non-blocking filesystem-level lock *and* a SQL level lock (for when we do multi-frontends). The filesystem level lock will keep case 1 from happening, and the SQL level lock will ensure frontend consistency.

We could also keep running the tasks in parallel *but* wait for them to finish before quitting the dispatcher. That way we easily control the number of parallel tasks running (which fixes #2). Services reload could also be offloaded to the dispatcher, which would restart apache/bind/whatever only once per dispatch run (instead of once per task right now).

Since this relates to self-hosting, i'm tagging this accordingly, see #454312: self-provisionning support.

I have made it so the dispatcher respects the "hosting_queue_NAME_running" variable (where NAME is the name of the queue): if it's set, it will avoid running the queue, so this variable can be considered a semaphore, to some extent (variable_set()/variable_get() isn't a atomic locking mechanism).

(We could use 6.16+'s lock.inc for this instead, but I'm not sure we want to depend on 6.16 just yet.)

Here's that first patch:

http://git.aegirproject.org/?p=hostmaster.git;a=commitdiff;h=6632d0a6faf...

Then I also made sure no more than N tasks are running in parallel, by checking already running tasks:

http://git.aegirproject.org/?p=hostmaster.git;a=commitdiff;h=406a7a0789a...

Finally, I made it so that the "PROCESSING" status of a task is set only when the task actually starts, instead of when the dispatch starts it, because the task may fail to start for unrelated reasons (the load check in provision_init() is a frequent example):

http://git.aegirproject.org/?p=hostmaster.git;a=commitdiff;h=9ab840f4b99...

I opened an issue about proper locking in #814296: use Drupal's semaphore/locking system instead of variable_get/set for dispatcher locking.