When the queue daemon crashes, we can end up with a LOT of "delete backups" tasks. We shouldn't create a new one for a site if one is still pending for the site, or better yet we should cancel the old one.

Patch incoming.

CommentFileSizeAuthor
#8 hosting_site_backup_manager-delete_backups_tasks-2687229-8.patch800 bytessteven jones

Comments

gboudrias created an issue. See original summary.

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
commented
Related issues: +#2445985: Failed backup "clogs up" the queue

Perhaps this is a duplicate?

gboudrias’s picture

gboudrias commented

They're both from me, I consider them separate problems.

This issue is about the delete backups task and the other one is about the backup task, which should be handled differently.

It's true that we need to never clog up the queue but I think the solution will be different for each, I doubt we can have a greater solution (forbid duplicates) as that would take away from the advantage of having versioned tasks (and the knowledge that your version of the task will run eventually).

Of course I'm open to all solutions. In this case I was about to patch it so as to cancel the older "delete backup" task (since the new task will presumably encompass all backups).

gboudrias’s picture

gboudrias commented
Related issues: +#2694037: Forbid duplicates of certain task types

After some thought I think we *can* have a larger solution: #2694037: Forbid duplicates of certain task types

In short, we should probably forbid duplicates of *some* task types.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
commented

An even better solution involves addressing "When the queue daemon crashes", which just shouldn't be happening. While certainly out-of-scope for this issue, I thought I'd mention some interesting projects that helmo pointed out in yesterday's scrum, that parallel Skynet somewhat: https://github.com/MartijnBraam/drush-queued, https://github.com/MartijnBraam/hosting_realtime_queue and https://github.com/MartijnBraam/sse-proxy.

gboudrias’s picture

gboudrias commented
Related issues: +#2672530: Adopt Python queue daemon replacement

Agreed. I'm adding #2672530: Adopt Python queue daemon replacement as a related issue, I think we should discuss it more there.

steven jones’s picture

steven jones commented
Assigned: Unassigned » steven jones

I've just run into this, and the queue ended up with 300,000 backup delete tasks in it before anyone noticed!
Aegir is now taking quite a long time to simply work out which task to run next, let alone run the task!

Anyway, seems like this should be a simple fix for this module and then something better can come later :)

steven jones’s picture

steven jones commented
Assigned: steven jones » Unassigned
Status: Active » Needs review
StatusFileSize
new hosting_site_backup_manager-delete_backups_tasks-2687229-8.patch800 bytes

Something like this should do the trick.

helmo’s picture

helmo commented
Status: Needs review » Reviewed & tested by the community

Looks good

  • helmo committed 30df244 on 7.x-3.x authored by Steven Jones 
    Issue #2687229 by Steven Jones: "delete backups" tasks clog up the task...
helmo’s picture

helmo commented
Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.