There are modules like backup queue that allow for tasks to run in background. I have also used the task queue to implement a few tasks as part of a custom development workflow (e.g. git pull). With a number of team members and lots of sites & platforms I have an aegir setup that is keep busy constantly running tasks during the day often a few tasks waiting in the queue.

There are tasks (like backup/migrate/clone) that can be time consuming/intensive on the server that can block the task queue for other less intensive tasks (git pull - which might take a few seconds to run). Some tasks are also less of a priority (backup) which is fine if they keep getting pushed to when the task queue is quiet.

I am looking to build a feature that will allow for tasks to be weighted and run based on a priority. The idea is for this to be configurable for a system, so depending on your requirements you could set the install's to have a high priority and it would mean that install tasks would always get put into the top of the queue and run ahead of a backup or other lower priority tasks.

Files: 

Comments

anarcat’s picture

The only problem I would see with such a system is that we loose the assumption for the user that tasks are ran serially. For example, a user may queue a "backup" job before a (say) "migrate" job, and assume they would run in that order. How would we deal with that assumption?

univate’s picture

My thought is that by default all tasks will have a weight of 0 and run serially as it does now.

But through a configuration page you would be able to drag task types around to increase their priority - so you would have to actually choose to make a change to the order. This would not be something coded into the task definition.

The implementation would be a new table with the list of tasks types and weights which when the queue runs would join across to pull out tasks based on weight.

I am trying to think of a good example where it would cause a problem (something like backup before migrate might be an issue if migrate didn't do backup first itself - I think there are enough protections in Aegir that it shouldn't cause a problem that can't be fixed) - interested if anyone can think of a use case that will break something.

anarcat’s picture

Fair enough - if it's optional, it makes sense. But it will make the task list a bit more confusing - make sure the task list reflects the task priorities. I think even now the task list display doesn't show clearly the queue...

ergonlogic’s picture

Version: 6.x-2.x-dev » 7.x-3.x-dev
Steven Jones’s picture

So I was bitten by this yesterday:

We have a platform with 600+ sites on it, and we migrated them from one platform to another.
Part of this process was to perform a backup on all of these sites, and the amount of disk space used grew and grew.
We do have a backup garbage collection system, but it couldn't run its tasks because the queue was running the previously queued migration tasks, which were just trying to use more and more disk space :(

I implemented a quick fix to select non-migration and non-backup tasks first, and then the queue processed normally and the server recovered.

I wonder if we can't use lovely D7 database layer to tag the query that gets the 'next' tasks to run so that other people can hook_db_query_alter the query to alter the definition of what the 'next' task is. That way, we can handle the weights and whatnot in contrib.

helmo’s picture

Steven Jones’s picture

Assigned: Unassigned » Steven Jones

Going to pick this one back up, since it's now totally possible to implement fairly easily.

Rough plan of attack:

  1. Refactor _hosting_get_new_tasks so that it uses a D7 style db_select and add a query tag to the query.
  2. Write a new module that adds priorities to tasks and stores them in a DB table.
  3. Hook into the query in 1 so that the 'next tasks to run' are influenced on task priorities too.
  4. Provide an hook implementation for setting default task priorities, and maybe a UI for tweaking it.
  5. When a new task node is created, get the default task priority and save it.
  6. Allow some UI for overriding task priority at task creation time. This might be useful if you have a lot of tasks running, but you really want a specific verify task to run before others, you could just set that single invocation of the task to urgent or similar.

I suspect I'll do 1 in the hosting module, and then spin the rest out into contrib for now.

ergonlogic’s picture

Sounds like a solid plan.

Steven Jones’s picture

Status: Active » Needs review
FileSize
3.25 KB

Here's a patch that does part 1, allows the query for 'get me next task' to be altered. I think that that'll be all the changes that are required in Aegir core.

These changes are also in the dev/1895580-task-priorities git branch.

ergonlogic’s picture

Status: Needs review » Needs work

Perhaps we could document the availability of the tag by adding a hook_query_TAG_NAME_alter() example in hosting.api.php.

Code looks good, but the implementation in contrib will be useful in testing this. Please post a link to that once it's ready.

Steven Jones’s picture

FileSize
4.14 KB

Thanks very much for the review.

Additional documentation for the hook added in the attached patch (and committed to git).

Leaving as Needs Work until the contrib module is ready.

Steven Jones’s picture

See https://www.drupal.org/sandbox/darthsteven/2730949 for the contrib part of the plan in #7.

Currently points 2, 3 and 5 are implemented. Point 4 is sort of covered in code, where delete and backup delete and login reset tasks are set as high priority.

Tested and working quite nicely :)

Steven Jones’s picture

Status: Needs work » Needs review

Points 4 is done too now, and point 6 has a firm plan in #2731013: Priority at task view time

This should be ready for review again.

Steven Jones’s picture

Assigned: Steven Jones » Unassigned

All the bits are there in the sandbox module now.