In 379d1edd, I fixed the task queue AJAX calls, so that they refresh the Views-based block. This may need back-porting to 6.x-2.x.

In doing so, I saw some room for improvement. First off, we set a refresh timeout of 30 seconds. It'd be nice to make this configurable. In addition, I'd like to look into shortening this timeout whenever there's a processing or queued task. Currently, when we trigger a task, it'll appear queued and then 30 seconds later the page refreshes, and in most cases the task is already complete.

Personally, this causes me to refresh the page quickly to see that the queue is processing the task. I often do so repeatedly when a task is processing, to see when it completes. I'm possibly just very impatient... But it seems like we could pass back a short timeout parameter in such cases, and then return to the default (or configured) longer period when the queue is idle.

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

helmo’s picture

*I'm just as impatient*

We currently hard code the 30000 value twice in task/hosting_task.js

So we could add a timeout setting to $settings['hostingTaskRefresh'] in hosting_task_preprocess_views_view_table()?

What about setting the timeout to 2 seconds after a task has started, and increasing it by a second every refresh until it reaches the max.

ergonlogic’s picture

I've been giving this some thought, and I think there are a couple issues.

  1. First, unless I'm mistaken, we're fully bootstrapping the entire site on every AJAX call. This includes Views, since we're replacing the view on every refresh. My concern is increasing the load on the front-end (by making AJAX calls more frequently) right when the server is under load from backend tasks. I haven't looked closely, but I suspect we can reduce the overhead here by:
    1. Adding file parameters to our menu items ('file' => 'xyz.admin.inc'), and moving everything we can out of .module files, so that they're only included when required. This has benefits beyond this issue.
    2. Consider adding the js.module to allow us to avoid full bootstraps on AJAX callbacks.
  2. Secondly, as I mentioned already, we're replacing the view on every refresh, regardless of whether there's been any change or not. Perhaps we could split this into two AJAX calls:
    1. The first be similar to the refresh call we make now, but not triggered on a set time-frame.
    2. The second would be a frequent polling to determine whether a change has occurred, which could then trigger a refresh when appropriate. For this lighter poll, the callback could just return the timestamp from the most recently added/updated task, since this should be a good indicator whether a refresh is required. We might also want to pass in the status of said task, to determine how frequently to poll.

I think 2.2 would more or less require 1.2, in order to really lighten up the processing required for each call. With something like this in place, I don't think we'd have any problem making polling calls every second or two when a task is queued or processing, and every 5-10 seconds otherwise. Of course these settings should still be configurable.

Also, I think 1.1 would make the polling from the queue daemon more efficient too.

ergonlogic’s picture

Commit cfe19c4 in the dev/2212149 branch makes the refresh rate configurable.

ergonlogic’s picture

Status: Active » Needs work

Commit a2ea5bb in dev/2212149 adds a callback that returns just the latest task timestamp and whether there are any outstanding tasks. We then poll that via AJAX, and refresh the queue block only when needed. Javascript (and jQuery) and are not my strong suit, so it'd be great to get some other eyes on this. The same treatment for the node task list will be required once we've got this working properly.

It mostly works, except that it won't update the last queued task to show it as completed. I think the JS is correct and I've documented the hell out of it, to try and make the logic clear. I believe the problem here is a bug in how we update tasks. Currently, all tasks show an 'Execution time' of '0 sec', and the 'executed' timestamp in hosting_task always matches the 'changed' timestamp in node. I think only the node has to updated at the completion of a task, but not both.

ergonlogic’s picture

Status: Needs work » Needs review

Still on the dev/2212149 branch, commit 79d8fd18 makes it so that we have node revisions when tasks are queued, executed and completed, as opposed to just updating the second on task completion. This allows us to know with confidence when a task status changes, as opposed to timestamps which were getting overwritten, and are only granular to the second. Commit 8a84091 adjusts the AJAX to compare vids and keep track of changed to the status across calls to the AJAX callbacks.

This appears to work properly, though there may be an extra poll or two after final tasks complete. Again, review by someone with more JS experience would be great.

Also, performance has already improved, if only slightly:

ab -n20 -c1 http://aegir3-dev.aegir3.local/hosting/tasks/queue
Requests per second:    7.15 [#/sec] (mean)
Time per request:       139.907 [ms] (mean)

ab -n20 -c1 http://aegir3-dev.aegir3.local/js/hosting_task/latest/status/
Requests per second:    8.89 [#/sec] (mean)
Time per request:       112.513 [ms] (mean)

Next, I'll see what improvement js.module can bring.

ergonlogic’s picture

Adding js.module to the mix results in the expected huge performance improvement:

ab -n20 -c1 http://aegir3-dev.aegir3.local/js.php?q=js/hosting_task/queue_status
Requests per second:    43.57 [#/sec] (mean)
Time per request:       22.954 [ms] (mean)
ergonlogic’s picture

One thing of note about admin_menu.module and js.module, the "Cache menu in client-side browser" setting appears to cause admin_menu to fail to load on certain pages. Disabling this setting seems to fix it though.

ergonlogic’s picture

Deploying js.module via makefile is a bit tricky, since we need a copy of js.php in the platform root. This snippet appears to work:

api=2
core=7

projects[] = "drupal"
libraries[js][download][type] = "file"
libraries[js][download][url] = "http://drupalcode.org/project/js.git/blob_plain/HEAD:/js.php"
libraries[js][download][filename] = "../../../../js.php"

Resulting in:

$ drush make js.make test && grep JS_MENU_NOT_FOUND test/ -rn
Beginning to build js.make.                                                      [ok]
drupal-7.26 downloaded.                                                          [ok]
js downloaded from http://drupalcode.org/project/js.git/blob_plain/HEAD:/js.php. [ok]
test/js.php:17:define('JS_MENU_NOT_FOUND', 2);
[...]

Note that it leaves an empty 'sites/all/libraries/js/' directory, where we might want to drop our own readme or something, explaining the where the file was moved to and why.

Steven Jones’s picture

Just to check, does that now skip any access checks, or are they still in place with the js.module ?

ergonlogic’s picture

js.module supports access checks, but it then needs to bootstrap to DRUPAL_BOOTSTRAP_SESSION instead of DRUPAL_BOOTSTRAP_DATABASE. This would reduce the performance gain, though I haven't checked by how much. Since the only data being returned is the latest task vid (an integer) and whether there are any outstanding tasks (a boolean), I don't think access checks are required here. Let me know if you think otherwise.

For the ajax refresh of the queue block itself nothing has (yet) changed. It still goes through the regular access checks. Since we build it with Views, and would need to keep access checks (and maybe i18n too), I don't know how much of a performance gain we'd see by putting it through js.php. The ajax appending to the task log seems like a better focus, and we'd obviously want access checks there too.

Steven Jones’s picture

Ah okay, was just want to make sure is all.
I guess the information disclosure is minimal and not really a security risk at all.

ergonlogic’s picture

It's definitely worth making that explicit :) I've added some documentation to our hook_js() implementation to explain why we skip access checks.

ergonlogic’s picture

In #2223063: Only call task confirm form when required I ajaxified the task buttons that don't require a confirmation, to simply refresh the task list, rather than load the overlay and reload the page right away. However, this introduced a regression in this issue, since we no longer refresh the page at all, and thus don't set the short polling timeout. I suspect that ajax_command_settings() is the solution.

helmo’s picture

Status: Needs review » Needs work

just tried, and looks great...

Not sure if this relates to #13, but... this sequence fails.

  1. Run a task which opens a dialog e.g. git pull, wait ... the increased poll frequency works OK.
  2. Run a task without a dialog, e.g. verify --> I get plain json in my main window.

  • Jon Pugh committed 5318d84 on 2212149-tasks-refresh-config
    Issue #2212149: Make task queue refresh configurable.
    
Jon Pugh’s picture

Status: Needs work » Needs review
FileSize
2.1 KB

I just pushed a new branch with a new approach... I think we should start with much smaller bites.

branch 2212149-tasks-refresh-config

All it does is replace hard coded 30000 with Drupal.settings.hostingTaskRefresh.refreshTimeout, and adds a hook_init() to add a drupal variable as a javascript variable. It currently defaults to the existing setting of 30000 to make it even easier to commit.

I suggest we commit this now, then get to work on adding a simple setting to the hosting config forms.

I think we should defer the work here on making a more efficient queue, and start something from scratch using modern APIs and something like ember or react.

Jon Pugh’s picture

As an aside, I was reminded this morning of "Vue.js". I was told it's the simplest js framework ever...

And well, in about an hour, I was able to put together an HTML template and Vue "app" for simulated aegir tasks:

https://jsfiddle.net/okv0rgrk/6100/

  • helmo committed 88370fd on 7.x-3.x authored by Jon Pugh
    Issue #2212149 by Jon Pugh: Improve task queue AJAX refresh
    
helmo’s picture

Status: Needs review » Needs work

Committed, except for the 'console.log' line ;)

Back to needs work for the next steps I guess...

Steven Jones’s picture

Can we just clean this up a little further please?

  • Define the JavaScript file and setting in a hook_library implementation.
  • Attach the library in hosting_task_view instead of calling drupal_add_js

hook_inits are pretty evil things.

Jon Pugh’s picture

I'm not against using libraries if it proves useful, but I warn against premature optimization/organization when this script needs so much love in other ways.

Perhaps once we create a better, modern javascript implementation we can move towards libraries?

Also note that this one javascript file contains the logic for the task block, the task node pages, and the ref-node pages (sites/servers/platforms), AND the hosting task confirm form which is why I went ahead and added it to hook init...

Which reminds me, we have unnecessary drupal_add_js() laying around now.

Steven: What about hook_init() is evil, given it's only being used to add javascript? Would it be "better" to attach it to a hook_page_alter() or something?

I am about to push a new patch that removes the extra drupal_add_js()

Jon Pugh’s picture

Patch and branch removes redundant drupal_add_js() calls:

11494427-hosting-task-js-remove

Jon Pugh’s picture

Status: Needs work » Needs review
Steven Jones’s picture

@Jon Pugh yeah, it's the fact that hook_init runs always, even if the request isn't going to return HTTP. So, when you run a Drush command the code gets called and the JavaScript setting is ready to go for the command line :)

Jon Pugh’s picture

Ah, that's a good reason!

So I did some digging, looks like the "scripts[]" line for info files works just like in themes: http://drupal.stackexchange.com/questions/132209/how-to-include-custom-j...

But that doesn't allow us to set variables... going to research the "right" way to add settings to all pages.

Jon Pugh’s picture

Drupal.org documentation recommends using hook_preprocess_page().

https://www.drupal.org/node/304255

helmo’s picture

Status: Needs review » Needs work
Jon Pugh’s picture

Assigned: Unassigned » Jon Pugh
Status: Needs work » Active

Let me see what I can whip up.

  • Jon Pugh committed 04a4bdd on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
  • Jon Pugh committed 078cd39 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
  • Jon Pugh committed 0d020a7 on 2212149-tasks-refresh-config
    Issue #2212149 by Jon Pugh: Improve task queue AJAX refresh, remove old...
  • Jon Pugh committed 4cbe547 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
  • Jon Pugh committed 58d6121 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
  • helmo committed 88370fd on 2212149-tasks-refresh-config authored by Jon Pugh
    Issue #2212149 by Jon Pugh: Improve task queue AJAX refresh
    
  • Jon Pugh committed a94e2e9 on 2212149-tasks-refresh-config
    Issue #2212149 by Jon Pugh: Improve task queue AJAX refresh, remove old...
  • Jon Pugh committed f7ce315 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...

  • Jon Pugh committed ee239e6 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
Jon Pugh’s picture

Hey, look at that. Drupal.org found the branch I'm using.

See 2212149-tasks-refresh-config.

  1. Vue.js is awesome. It basically just takes a javascript data object and let's you bind it to certain elements in the DOM, with tools for looping and all sorts of things.
  2. Hosting Task block is now dynamicly updated anytime the object Drupal.settings.hostingTasks.tasks is updated.
  3. I hacked in a custom template for the Views Table to allow myself to add the vue.js markup. This means this block is no longer customizable in the views interface (for now).
  4. I changed hosting/task/queue to return JSON of the hosting tasks instead of the rendered view.
  5. The end result is identical to the existing block.
  6. I commented out all of hosting_task.js to start from scratch, so the other blocks are not working yet.
  7. We should map all of our existing objects to basic data structures that can then be bound to the markup. Think of all the properties on platforms, sites, etc. Those can update in the background. If vue.js is wired in, those lists would update in real time.

  • Jon Pugh committed d133c6f on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...

  • Jon Pugh committed 8b397e2 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
  • Jon Pugh committed b16e3dc on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...

  • Jon Pugh committed 293f21a on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
  • Jon Pugh committed fd15e79 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
Jon Pugh’s picture

FileSize
26.49 KB

Patch attached for your review, or see branch of same name.

Still TODO:

  1. Clean up "task types" visibility issues. Clarify "task_permitted" vs hidden vs actually not available.
  2. Docblocks the functions we are improving that were always lacking documentation.
  3. Add to full page view.
  4. Setup per-user caching of the JSON results with cache busting happening on task node update or creation.
  5. Add vue.js as a library instead of loading it remotely.
  6. Add node view fields. Now that the available tasks update automatically, we should update the list of displayed fields too!

  • Jon Pugh committed b4040d7 on 2212149-tasks-refresh-config
    Issue #2212149: Improve task queue AJAX refresh. Progress in...
Jon Pugh’s picture

FileSize
37.85 KB
Jon Pugh’s picture

FileSize
691.13 KB

In this last round of fixes, I've also made some improvements to the task block UI:

  1. Added ref type name. "Verify Platform", "Verify Site"
  2. Added a timestamp.
  3. Added timeago plugin so it updates automatically.

The reason some tasks are missing is because there is confusion about when to show a task. "hidden", "task_permitted", and presence in the list are all confused.

I'll keep working on it.

Jon Pugh’s picture

Close up of the task queue block:

  • Jon Pugh committed 68bca3f on 2212149-tasks-refresh-config
    Issue #2212149: Output static data into "noscript" tags so we have...
Jon Pugh’s picture

Status: Active » Needs review
FileSize
23.87 KB

Just pushed more improvements:

  • Added a views field handler for "Task Type", allowing configuration of raw vs text version.
  • Added a function hosting_task_type_name() to allow for easy retrieval of task type titles.
  • Removed the extra field on Tasks views that always shows the node reference type in favor of...
  • Changed the task titles for all tasks that have the same name across types: Verify Site, Verify Platform, Verify Server, Delete Site, Delete Platform, Delete Server.
  • Fixed reattaching of timeago javascript so new tasks always start at 0 seconds.
  • Added static output inside tags so it still works without javascript.
  • New screenshot of task queue: Note only see "Site" and "Platform" for tasks where it's needed to clarify: "Verify Platform", "Verify Site".

Jon Pugh’s picture

Final TODOs before ready to merge:

  1. Implement server side caching of tasks data.
  2. Implement Vue.js for global tasks page.
  3. Implement dynamic loading of Aegir Object properties.
  4. Add timeago.module, which allows us to use a drupal date format for dynamic "time ago" and would allow us to not manually include jquery.timeago.js
Jon Pugh’s picture

Patch attached to include timeago.module and jquery into hostmaster. Branch available in hostmaster project as well: 2212149-tasks-refresh-config

colan’s picture

helmo’s picture

Was changing the order of the task buttons intentional here?
Anyway it's a request in #2821741: Rearrange task buttons

clemens.tolboom’s picture

I don't like the dependency to http://npmcdn.com which relays to https://unpkg.com.
- Having NoScript this leads to "Vue is not defined" (doh)
- There is no need to expose referrer right?

The polling runs every 2 secs which is ok when one is waiting for a task to complete but the polling continues even without active tasks which is a waste of server load.
- is 2 secs hard coded? Could is be delay back to 30 secs?
- having a few windows open gives substantial load on my server.

my 2 cents

helmo’s picture

+1 for getting vue in the makefile ... although we will have to get it on the https://www.drupal.org/project/drupalorg_whitelist first

Another thing ... I get this notice:
Notice: Undefined property: stdClass::$hidden in include() (line 19 of /data2/aegir/hostmaster-7.x-3.x-2016-10-25-1529/profiles/hostmaster/modules/aegir/hosting/task/templates/hosting-task-table.tpl.php)

Jon Pugh’s picture

Thanks for the feedback, all.

I don't like the dependency to http://npmcdn.com which relays to https://unpkg.com.

Agreed, I did this out of laziness to get started quickly. Then I forgot to add that to the TODO list. :) With most things I make sure to include all code, I host many devshops in private networks.

Having NoScript this leads to "Vue is not defined" (doh)

Hmm, I haven't seen this yet. How would that run if there's no JS? Could you provide a URL and maybe a screenshot?

Was changing the order of the task buttons intentional here?

No. The code that generates the list used to be the code that generated the static table, so i thought it would be the same.

The polling runs every 2 secs which is ok when one is waiting for a task to complete but the polling continues even without active tasks which is a waste of server load.
- is 2 secs hard coded? Could is be delay back to 30 secs?
- having a few windows open gives substantial load on my server.

This uses the same variable that the existing tasks uses, I just changed the default to 2000. I'll post an updated patch.

The reason for continued polling is that we want new tasks to appear as soon as possible. There is no other way for the server to let browsers know about new tasks without adding a lot: new process to the server for websockets or one of those other systems.

In the TODOs above is caching. We can ensure optimal performance by saving the JSON payload into a cache, and using a hook_node_update() or hook_node_insert() to trigger cache clear. We might want to consider rebuilding the cache in the node update, so the web requests don't have to do it. It will have to be cached per user, but that should be good.

If we want further improvements we can implement "long polling", meaning we increase (or turn off) PHP timeout for the request and keep the request open until there is a new task. I actually do this to enable the "login" button in devshop to only need one "click". See the hook_menu callback "devshop_hosting_login_reset": https://github.com/opendevshop/devmaster/blob/1.x/modules/devshop/devsho...

This however, simply transfers the "polling" to the server side. The difference is we can do a more efficient query and we are only bootstrapping drupal once.

For users that don't care about immediate updates, we should add a settings form so they can set their task updates to longer or even turn it off if they want.

  • Jon Pugh committed 6a8da29 on 2212149-tasks-refresh-config
    Issue #2212149: Improving Task AJAX
    - Added update to enabling timeago...
Jon Pugh’s picture

New patch attached with new improvements:

  1. Added update to enabling timeago module and save date format.
  2. Added hook hook_date_format_types() to enable a "Dynamic Time Ago" format.
  3. Exporting the hosting_task_list view to use the Dynamic Time Ago format.

Still have to make those views dynamic,however I am feeling like a more generic vue.js views module would be more appropriate than another "static" views template...

I don't want perfect to be the enemy of good, though.

Jon Pugh’s picture

Neograph734’s picture

Was changing the order of the task buttons intentional here?

No. The code that generates the list used to be the code that generated the static table, so i thought it would be the same.

The order of the elements in HTML has not changed, but because of a float: right the first element was at the far right. But as I explained in #2821741: Rearrange task buttons, I think it makes more sense this way. Please keep it :)

clemens.tolboom’s picture

Having NoScript this leads to "Vue is not defined" (doh)

Hmm, I haven't seen this yet. How would that run if there's no JS? Could you provide a URL and maybe a screenshot?

Only the remote JSs where blocked. Hostmaster was allowed.

Jon Pugh’s picture

helmo’s picture

Status: Needs review » Needs work

so the remaining todo's:

  • get rid of the remote js
  • fix timeago error, see below.
  • ...

I just got these in my JS console, while admin/reports/status mentions that the timeago lib is OK:

You are running Vue in development mode.
Make sure to turn on production mode when deploying for production.
See more tips at https://vuejs.org/guide/deployment.html

vue.js (line 6109)
TypeError: $.timeago is undefined
helmo’s picture

I've put in a #2847031: Request to add vue.js to the packaging whitelist.

helmo’s picture

Yay, vue is now on the whitelist ... please add it to the makefile.