PHP is know to be a bad choice for writing daemons. Anyone who bothers to read our hosting_queued code will see that the majority of it is made up of shims to work around PHP's limitations in this regard. Python, on the other hand, is very well-suited to writing daemons. As such, I wrote a very simple daemon, based on a Python CLI framework called Cement, along with (PHP) Drush/Provision hooks to write out its config file containing DB credentials, and a Puppet module to install it: https://github.com/GetValkyrie/skynet.

I suggest that we adopt this as an alternative queue daemon.

Comments

ergonlogic created an issue.

gboudrias’s picture

gboudrias CreditAttribution: gboudrias at Praxis Labs Coop commented

+1, I've used Skynet in prod (in 6.x) and will probably switch to it again soon regardless (the PHP daemon has crashed on me a few times even in 7.x).

helmo’s picture

helmo CreditAttribution: helmo at Initfour websolutions commented

Even though the php queued has improved a bit in recent versions this looks like a promising way to go forward.

I suggest we combine a few things here... and call the repo hosting_skynet.

Contents:
- https://github.com/MartijnBraam/hosting_realtime_queue
- https://github.com/ergonlogic/skynet (as a base for the Drush integration and credentials handling)
- https://github.com/MartijnBraam/drush-queued (daemon code and event source)
- Service subclass, see #2714065: Make the task queue a service

This was we can add the module as a Golden Contrib to our Distribution makefile.

Can we automate the needed apache/nginx config to pass on the /eventsource url?

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented

I'd really like to see this move forward as I'm growing somewhat annoyed at the conventional queue runner dying.

Are the MartijnBramm projects very different from ergonlogic's work? ergonlogic tells me he's been running it without problems though Ansible. Maybe start there, and add additional things later?

Ideally for now, we'd simply enable an Experimental module in Golden Contrib, and it would shut down the conventional runner and start itself up (after following some preliminary set-up instructions (for prerequisites, etc.). Even not having it in Golden Contrib would be fine for now. I'd just like something easy to follow that won't mess up Debian package management. Maybe we just need to polish the Skynet README, or use it as-is.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic at Ergon Logic Enterprises for Aegir Cooperative commented

As far as I can tell, https://github.com/MartijnBraam/hosting_realtime_queue is meant to replace the JS polling of the task table, and replace it with a web-socket that's subscribed to events coming from https://github.com/MartijnBraam/drush-queued. Since both poll the database for tasks, and then execute 'drush @hostmaster hosting-task', we could certainly merge them. I think Martijn's is probably a better base, as it'd allow us to drop Cement as a dependency (which I'm not leveraging much anyway).

I *think* we should be able to create a single project that includes all the components, and would allow it to be deployed as a front-end module, thus making it easy to add to the hostmaster makefile.

https://github.com/ergonlogic/hosting_skynet is the current iteration of the queued replacement that I've been using. Among other things, it'll generate the credentials file that is required to poll the task table. https://github.com/MartijnBraam/drush-queued also needs such credentials passed in, so I think we should take that bit from skynet.

Martijn's project includes a systemd service file, whereas hosting_skynet includes a Supervisor config file, to allow the service to be logged and restarted as needed. I think a small install.sh script is probably in order to simplify installation (similar to the permission/ownership modules in hosting_tasks extra).