Implement a system log for Aegir operations

The point would be to have a system log that could be parsed by something like logstash. In the IRC exchange below I suggest a couple implementation options for such a log. I think the best solution is to implement our own DRUSH_LOG_CALLBACK (see drush_invoke_process()). We'd still need to call _drush_print_log(), so that the output still gets
passed back to the front-end. But it seems like it'd be pretty straight-forward to the just add a time-stamp and write out through syslog, or something similar.

15:52      xur1z > something i often wonder, but only for as long as it takes for tasks to complete so i never get around to checking in depth
15:52      xur1z > is there a Drush command for Aegir tasks, equivalent to drush watchdog-show --tail
15:52      xur1z > OR
15:53      xur1z > can i enable drush task info outputting to watchdog, which would achieve same
17:24      xur1z > realityloop, do you ever monitor tasks from console, and if so how? (see Q above)
17:24      xur1z > gut says it's there and i've just not looked in the right place to enable it
17:25      xur1z > (but i only ever think to look of it momentarily, while waiting on task to finish, so i never dig)
[...]
11:14 ergonlogic > xur1z: In Aegir 3.x the task log should update via ajax
11:14 ergonlogic > if you run the task manually, you should see the Drush output
11:14 ergonlogic > but otherwise it's captured in the Drush process, and piped back into the front-end
11:15 ergonlogic > I sometimes monitor the state of ongoing tasks using htop, which will show the currently running process tree
16:22      xur1z > ergonlogic huh, interesting. so then is it that the drush call is running outside the context of the site, and can't watchdog() 
                   to the correct location?
16:22      xur1z > guessing there is an architecture reason why we don't see log output, else we'd see some
16:24 ergonlogic > you could write to watchdog as part of a drush command run on a site, but we haven't implemented anything like that
16:24 ergonlogic > I'm not sure why you'd want to
16:25 ergonlogic > but we already capture the drush output and write it to the task log
16:29 ergonlogic > xur1z: I may be missing the point of your question
16:31      xur1z > i just tend to sit on logfiles for progress indication
16:31      xur1z > so even with pretty ajaxy throbbers, i want to see where it's up to :)
16:32      xur1z > i'll try hooking into the provision commands and see if i get watchdog output - thanks. totally just a for-interest thing, and to 
                   keep me reassured while tasks run
16:34 ergonlogic > so, it's probably somewhere in the implementation of the hosting-task Drush command
16:35      xur1z > ok
16:36 ergonlogic > xur1z: http://api.aegirproject.org/api/aegir/hosting3%21task.hosting.inc/function/drush_hosting_task/7.x-3.x
16:37      xur1z > cheers
16:41 ergonlogic > so you could probably implement a drush_post_hosting_task() and look in drush_get_context('HOSTING_DRUSH_OUTPUT')
16:41 ergonlogic > presumably you could write that to a log file
16:42 ergonlogic > it won't exactly be in realtime
16:42 ergonlogic > it'd be lumped together by task
16:45 ergonlogic > xur1z: another option would be in Drush itself: http://api.drush.org/api/drush/includes%21drush.inc/function/drush_log/6.x
16:46 ergonlogic > it looks like you can set an alternate "drush_get_context('DRUSH_LOG_CALLBACK')"
16:46      xur1z > that'd just get back to the eventual command output, not trigger something to watchdog immediately? (i will explore this but 
                   can't right now because work!)
16:47 ergonlogic > so a better solution might be to intercept the output there
16:47 ergonlogic > right, you'd still have to handle writing it to watchdog
16:47 ergonlogic > which'd be complicated
16:47 ergonlogic > possibly
16:48 ergonlogic > I mean, you could write to the site's watchdog, but Aegir's watchdog would be a little tougher
16:49 ergonlogic > anyway, I'd just write a wrapper for _drush_print_log(), that writes output to a logfile first
16:49 ergonlogic > that way you could just tail the logfile directly
16:49 ergonlogic > since putting the output in watchdog is presumably not really necessary, as such
16:50 ergonlogic > actually, this is something I've been wishing for myself
16:50 ergonlogic > I'll write up a feature request