Improve task queue AJAX refresh

*I'm just as impatient*

We currently hard code the 30000 value twice in task/hosting_task.js

So we could add a timeout setting to $settings['hostingTaskRefresh'] in hosting_task_preprocess_views_view_table()?

What about setting the timeout to 2 seconds after a task has started, and increasing it by a second every refresh until it reaches the max.

Just to check, does that now skip any access checks, or are they still in place with the js.module ?

Ah okay, was just want to make sure is all.
I guess the information disclosure is minimal and not really a security risk at all.

just tried, and looks great...

Not sure if this relates to #13, but... this sequence fails.

1. Run a task which opens a dialog e.g. git pull, wait ... the increased poll frequency works OK.
2. Run a task without a dialog, e.g. verify --> I get plain json in my main window.

I just pushed a new branch with a new approach... I think we should start with much smaller bites.

branch 2212149-tasks-refresh-config

All it does is replace hard coded 30000 with Drupal.settings.hostingTaskRefresh.refreshTimeout, and adds a hook_init() to add a drupal variable as a javascript variable. It currently defaults to the existing setting of 30000 to make it even easier to commit.

I suggest we commit this now, then get to work on adding a simple setting to the hosting config forms.

I think we should defer the work here on making a more efficient queue, and start something from scratch using modern APIs and something like ember or react.

As an aside, I was reminded this morning of "Vue.js". I was told it's the simplest js framework ever...

And well, in about an hour, I was able to put together an HTML template and Vue "app" for simulated aegir tasks:

https://jsfiddle.net/okv0rgrk/6100/

Committed, except for the 'console.log' line ;)

Back to needs work for the next steps I guess...

Can we just clean this up a little further please?

• Define the JavaScript file and setting in a hook_library implementation.
• Attach the library in hosting_task_view instead of calling drupal_add_js

hook_inits are pretty evil things.

I'm not against using libraries if it proves useful, but I warn against premature optimization/organization when this script needs so much love in other ways.

Perhaps once we create a better, modern javascript implementation we can move towards libraries?

Also note that this one javascript file contains the logic for the task block, the task node pages, and the ref-node pages (sites/servers/platforms), AND the hosting task confirm form which is why I went ahead and added it to hook init...

Which reminds me, we have unnecessary drupal_add_js() laying around now.

Steven: What about hook_init() is evil, given it's only being used to add javascript? Would it be "better" to attach it to a hook_page_alter() or something?

I am about to push a new patch that removes the extra drupal_add_js()

Patch and branch removes redundant drupal_add_js() calls:

11494427-hosting-task-js-remove

@Jon Pugh yeah, it's the fact that hook_init runs always, even if the request isn't going to return HTTP. So, when you run a Drush command the code gets called and the JavaScript setting is ready to go for the command line :)

Ah, that's a good reason!

So I did some digging, looks like the "scripts[]" line for info files works just like in themes: http://drupal.stackexchange.com/questions/132209/how-to-include-custom-j...

But that doesn't allow us to set variables... going to research the "right" way to add settings to all pages.

Drupal.org documentation recommends using hook_preprocess_page().

https://www.drupal.org/node/304255

Let me see what I can whip up.

Hey, look at that. Drupal.org found the branch I'm using.

See 2212149-tasks-refresh-config.

1. Vue.js is awesome. It basically just takes a javascript data object and let's you bind it to certain elements in the DOM, with tools for looping and all sorts of things.
2. Hosting Task block is now dynamicly updated anytime the object Drupal.settings.hostingTasks.tasks is updated.
3. I hacked in a custom template for the Views Table to allow myself to add the vue.js markup. This means this block is no longer customizable in the views interface (for now).
4. I changed hosting/task/queue to return JSON of the hosting tasks instead of the rendered view.
5. The end result is identical to the existing block.
6. I commented out all of hosting_task.js to start from scratch, so the other blocks are not working yet.
7. We should map all of our existing objects to basic data structures that can then be bound to the markup. Think of all the properties on platforms, sites, etc. Those can update in the background. If vue.js is wired in, those lists would update in real time.

Patch attached for your review, or see branch of same name.

Still TODO:

1. Clean up "task types" visibility issues. Clarify "task_permitted" vs hidden vs actually not available.
2. Docblocks the functions we are improving that were always lacking documentation.
3. Add to full page view.
4. Setup per-user caching of the JSON results with cache busting happening on task node update or creation.
5. Add vue.js as a library instead of loading it remotely.
6. Add node view fields. Now that the available tasks update automatically, we should update the list of displayed fields too!

In this last round of fixes, I've also made some improvements to the task block UI:

1. Added ref type name. "Verify Platform", "Verify Site"
2. Added a timestamp.
3. Added timeago plugin so it updates automatically.

The reason some tasks are missing is because there is confusion about when to show a task. "hidden", "task_permitted", and presence in the list are all confused.

I'll keep working on it.

Close up of the task queue block:

Just pushed more improvements:

• Added a views field handler for "Task Type", allowing configuration of raw vs text version.
• Added a function hosting_task_type_name() to allow for easy retrieval of task type titles.
• Removed the extra field on Tasks views that always shows the node reference type in favor of...
• Changed the task titles for all tasks that have the same name across types: Verify Site, Verify Platform, Verify Server, Delete Site, Delete Platform, Delete Server.
• Fixed reattaching of timeago javascript so new tasks always start at 0 seconds.
• Added static output inside tags so it still works without javascript.
• New screenshot of task queue: Note only see "Site" and "Platform" for tasks where it's needed to clarify: "Verify Platform", "Verify Site".

Final TODOs before ready to merge:

1. Implement server side caching of tasks data.
2. Implement Vue.js for global tasks page.
3. Implement dynamic loading of Aegir Object properties.
4. Add timeago.module, which allows us to use a drupal date format for dynamic "time ago" and would allow us to not manually include jquery.timeago.js

Patch attached to include timeago.module and jquery into hostmaster. Branch available in hostmaster project as well: 2212149-tasks-refresh-config

Related: https://about.gitlab.com/2016/10/20/why-we-chose-vue/

Was changing the order of the task buttons intentional here?
Anyway it's a request in #2821741: Rearrange task buttons

I don't like the dependency to http://npmcdn.com which relays to https://unpkg.com.
- Having NoScript this leads to "Vue is not defined" (doh)
- There is no need to expose referrer right?

The polling runs every 2 secs which is ok when one is waiting for a task to complete but the polling continues even without active tasks which is a waste of server load.
- is 2 secs hard coded? Could is be delay back to 30 secs?
- having a few windows open gives substantial load on my server.

my 2 cents

+1 for getting vue in the makefile ... although we will have to get it on the https://www.drupal.org/project/drupalorg_whitelist first

Another thing ... I get this notice:
Notice: Undefined property: stdClass::$hidden in include() (line 19 of /data2/aegir/hostmaster-7.x-3.x-2016-10-25-1529/profiles/hostmaster/modules/aegir/hosting/task/templates/hosting-task-table.tpl.php)

Thanks for the feedback, all.

I don't like the dependency to http://npmcdn.com which relays to https://unpkg.com.

Agreed, I did this out of laziness to get started quickly. Then I forgot to add that to the TODO list. :) With most things I make sure to include all code, I host many devshops in private networks.

Having NoScript this leads to "Vue is not defined" (doh)

Hmm, I haven't seen this yet. How would that run if there's no JS? Could you provide a URL and maybe a screenshot?

Was changing the order of the task buttons intentional here?

No. The code that generates the list used to be the code that generated the static table, so i thought it would be the same.

The polling runs every 2 secs which is ok when one is waiting for a task to complete but the polling continues even without active tasks which is a waste of server load.
- is 2 secs hard coded? Could is be delay back to 30 secs?
- having a few windows open gives substantial load on my server.

This uses the same variable that the existing tasks uses, I just changed the default to 2000. I'll post an updated patch.

The reason for continued polling is that we want new tasks to appear as soon as possible. There is no other way for the server to let browsers know about new tasks without adding a lot: new process to the server for websockets or one of those other systems.

In the TODOs above is caching. We can ensure optimal performance by saving the JSON payload into a cache, and using a hook_node_update() or hook_node_insert() to trigger cache clear. We might want to consider rebuilding the cache in the node update, so the web requests don't have to do it. It will have to be cached per user, but that should be good.

If we want further improvements we can implement "long polling", meaning we increase (or turn off) PHP timeout for the request and keep the request open until there is a new task. I actually do this to enable the "login" button in devshop to only need one "click". See the hook_menu callback "devshop_hosting_login_reset": https://github.com/opendevshop/devmaster/blob/1.x/modules/devshop/devsho...

This however, simply transfers the "polling" to the server side. The difference is we can do a more efficient query and we are only bootstrapping drupal once.

For users that don't care about immediate updates, we should add a settings form so they can set their task updates to longer or even turn it off if they want.

New patch attached with new improvements:

1. Added update to enabling timeago module and save date format.
2. Added hook hook_date_format_types() to enable a "Dynamic Time Ago" format.
3. Exporting the hosting_task_list view to use the Dynamic Time Ago format.

Still have to make those views dynamic,however I am feeling like a more generic vue.js views module would be more appropriate than another "static" views template...

I don't want perfect to be the enemy of good, though.

Was changing the order of the task buttons intentional here?

No. The code that generates the list used to be the code that generated the static table, so i thought it would be the same.

The order of the elements in HTML has not changed, but because of a float: right the first element was at the far right. But as I explained in #2821741: Rearrange task buttons, I think it makes more sense this way. Please keep it :)

Having NoScript this leads to "Vue is not defined" (doh)

Hmm, I haven't seen this yet. How would that run if there's no JS? Could you provide a URL and maybe a screenshot?

Only the remote JSs where blocked. Hostmaster was allowed.

Broke out this improvement to another issue: #2823827: Specify object when tasks are used on may objects: "Verify Site" vs "Verify Platform"

so the remaining todo's:

• get rid of the remote js
• fix timeago error, see below.
• ...

I just got these in my JS console, while admin/reports/status mentions that the timeago lib is OK:

You are running Vue in development mode.
Make sure to turn on production mode when deploying for production.
See more tips at https://vuejs.org/guide/deployment.html

vue.js (line 6109)
TypeError: $.timeago is undefined

I've put in a #2847031: Request to add vue.js to the packaging whitelist.

Yay, vue is now on the whitelist ... please add it to the makefile.

Pushed updated makefile including vue.min.js to branch 2212149-tasks-refresh-config in hostmaster.

I did another review of this ...

timeago wasn't loading for me. I updated the makefile to place it in timago instead of jquery.timeago and added two calls to timeago_add_js();

I've set the default refresh frequency back from 2s to 20s... too much server load. Making that variable would be a nice follow up. Maybe begin at 2s and slowly increase when nothing is happening on the site.

Remaining TODO's:

• * replace `drupal_add_js('https://npmcdn.com/vue/dist/vue.js', 'external');` with the bundled version.
• * switch off vue dev mode?
  

  You are running Vue in development mode.
  Make sure to turn on production mode when deploying for production.
  See more tips at https://vuejs.org/guide/deployment.html

first (not working) attempt to solve my last todo.

vue seems OK now with my last commit.

One new TODO I found ... the task block now uses the page_global or page_contextual view display. And not the 'block' one. This results in the first load having 25 items, and after the first ajax refresh it goes back to 5.

I've merged the hostmaster part of this patch.

I'm not sure we should merge the hosting part before 3.13, but this will make testing easier for a larger userbase.

remaining issues list in the summary

The noscript tag is now used in a number of templates .... however it seems that it's not allowed inside a table tag. When JS is deactivated the task list remains empty. Do we really have to duplicate the table headers?

I'll poke around on this one again now.