There are modules like backup queue that allow for tasks to run in background. I have also used the task queue to implement a few tasks as part of a custom development workflow (e.g. git pull). With a number of team members and lots of sites & platforms I have an aegir setup that is keep busy constantly running tasks during the day often a few tasks waiting in the queue.

There are tasks (like backup/migrate/clone) that can be time consuming/intensive on the server that can block the task queue for other less intensive tasks (git pull - which might take a few seconds to run). Some tasks are also less of a priority (backup) which is fine if they keep getting pushed to when the task queue is quiet.

I am looking to build a feature that will allow for tasks to be weighted and run based on a priority. The idea is for this to be configurable for a system, so depending on your requirements you could set the install's to have a high priority and it would mean that install tasks would always get put into the top of the queue and run ahead of a backup or other lower priority tasks.


anarcat’s picture

The only problem I would see with such a system is that we loose the assumption for the user that tasks are ran serially. For example, a user may queue a "backup" job before a (say) "migrate" job, and assume they would run in that order. How would we deal with that assumption?

univate’s picture

My thought is that by default all tasks will have a weight of 0 and run serially as it does now.

But through a configuration page you would be able to drag task types around to increase their priority - so you would have to actually choose to make a change to the order. This would not be something coded into the task definition.

The implementation would be a new table with the list of tasks types and weights which when the queue runs would join across to pull out tasks based on weight.

I am trying to think of a good example where it would cause a problem (something like backup before migrate might be an issue if migrate didn't do backup first itself - I think there are enough protections in Aegir that it shouldn't cause a problem that can't be fixed) - interested if anyone can think of a use case that will break something.

anarcat’s picture

Fair enough - if it's optional, it makes sense. But it will make the task list a bit more confusing - make sure the task list reflects the task priorities. I think even now the task list display doesn't show clearly the queue...

ergonlogic’s picture

Version:6.x-2.x-dev» 7.x-3.x-dev
Steven Jones’s picture

So I was bitten by this yesterday:

We have a platform with 600+ sites on it, and we migrated them from one platform to another.
Part of this process was to perform a backup on all of these sites, and the amount of disk space used grew and grew.
We do have a backup garbage collection system, but it couldn't run its tasks because the queue was running the previously queued migration tasks, which were just trying to use more and more disk space :(

I implemented a quick fix to select non-migration and non-backup tasks first, and then the queue processed normally and the server recovered.

I wonder if we can't use lovely D7 database layer to tag the query that gets the 'next' tasks to run so that other people can hook_db_query_alter the query to alter the definition of what the 'next' task is. That way, we can handle the weights and whatnot in contrib.

helmo’s picture