Problem/Motivation

Drupal provides numerous built-in user interactions. These include back-end admin configuration screens, but also end-user components. For authenticated users with admin permissions, that includes components such as the Toolbar, Contextual Links and Quick Edit. For all users, Drupal provides components like comments, search and user login forms.

As a server/PHP-based content management framework, Drupal relies a great deal on full page refreshes to provide additional options after a user interacts with a component. Full page refreshes can sometimes be avoided through the usage of Drupal's home-grown AJAX tools or through included third-party tools like jQuery or Backbone. Many of these tools still require a good deal of behind-the-scenes server interactions.

Meanwhile expectations for user interactions have increased, both due to the responsiveness of mobile native applications and JS-based single-page applications. Google details these expectations through a performance model called RAIL: https://developers.google.com/web/tools/chrome-devtools/profile/evaluate....

  • Response: UI response received within 100ms of user interaction (16ms for touchdrag)
  • Animation and scrolling: Render frames every 16ms (60 fps)
  • Idle: Behind the scenes interactions with the server take place in batches of 50ms or less, to allow time for responding to user interactions
  • Load: Full page loads take place in less than 1 second even on mobile connections (could be stretched up to 3 seconds)

There could be other metrics for performance. If power management (battery drain) and memory impacts were taken into consideration, this might be called PRIMAL for example.

Boosting key performance metrics would certainly help improve the experience of using Drupal, but more than that is probably necessary. Removing full page refreshes in order to show the results of a user interaction would also help. This is often called Optimistic UI: for a good explanation, see http://info.meteor.com/blog/optimistic-ui-with-meteor-latency-compensation.

We may be able to provide additional boosts to perceived performance and optimistic UI additions in the Drupal 8 cycle: some may need to wait until Drupal 9.

Proposed resolution

The goal of this META is to look at possible ways to meet the stated goals and evaluate the pros and cons of each. Ideally we reach some degree of shared consensus on ways to move forward.

Big Pipe (https://www.drupal.org/project/big_pipe) is one initiative underway that will help boost perceived load times. That may not have an impact on back-end administration screens unless cache tags are used for back-end components, which may or may not be useful.

Client-side JS frameworks are one way that many sites improve response time, particularly by storing large chunks of data in memory (often through JSON) that can be used to provide responses without the latency required for server roundtrips. The downside is often that client-side frameworks typically require a render-blocking download of the JS framework before the initial HTML for a page can be generated. This slows down the L in RAIL, particularly on mobile.

Client-side frameworks have tackled that challenge through the use of isomorphic JS: the initial page render is handled with server-side JS, typically with the use of Node. This helps to provide the best of both worlds.

Taking advantage of that solution with Drupal might prove very challenging, however. Pure usage of client-side frameworks could mean that content would only render when JS is active and functioning error-free. Drupal has typically excelled at progressive enhancement and accessibility, and relying solely on client-side rendering might contradict those principles. Using server-side JS rendering could require a large overhaul of Drupal's theme system in order to work on Node. Rendering some components with server-side JS (using Node) and others with Drupal's typical server-side PHP may or may not be possible.

The JS framework landscape is also highly volatile right now. Because an overhaul of Drupal's theme system could be time consuming, any client side framework selected could lose momentum as new frameworks with new techniques emerge. We have seen that when new technique emerges, it is sometimes adopted by other frameworks. React improved response time with virtual DOM diffing and load time with isomorphic rendering, and now Ember and Angular in their latest versions are adopting their techniques. If a framework is used to improve Drupal's user experience, we may want to select one that is likely to stay up to date with emerging techniques while providing backwards compatibility and security updates if their update pace is faster than what Drupal can keep up with through our release schedule.

Newer PHP-based techniques that use long-running PHP daemons to provide faster server responses without full bootstraps could also be explored. That does not eliminate latency from server roundtrips, but might be useful to explore.

Whichever path is used to improve Drupal's user experience, improvements to Drupal's APIs will likely be required—particularly for administrative interfaces.

Key goals to keep in mind:

  • Fast initial load time: user interaction improvements ideally still use progressive enhancement
  • Fast responses to user interactions
  • Minimize full page refreshes and server bootstraps

Other issues have explored including client-side frameworks in core (#2645250: [META] Start using reactive declarative JS programming for some new core admin UIs) and moving all or part of Drupal's theme system to render with Node JS (#2645666: [policy, no patch] Require Node.js for a future Drupal major version and rewrite some of Drupal's UI code from PHP to JS). Those might be methods we end up exploring in order to meet our goals. The goal here is to take a step back and look more fundamentally at what are the user experience goals we want to achieve and what are the best ways to do so for the Drupal community.

Ideally we find options that excite the existing Drupal community about the direction we head and may even excite others outside the Drupal community about the front-end (and back-end) work being done to improve Drupal's user experience.

Remaining tasks

- Propose various ways user experience goals could be met
- Evaluate the pros and cons of each method
- Reach consensus on ways forward

User interface changes

Ideally this results in numerous user interface changes that simplify how users interact with Drupal.

API changes

API changes could range from improving Drupal's REST API for administrative interfaces to a complete rewrite of the theme system.

Data model changes

In theory Drupal's data model should not need to change.

Comments

mdrummond created an issue. See original summary.

RainbowArray’s picture

Title: [META] Dramatically improve Drupal-provided user interactions through perceived performance boosts and optimistic user interactions » [META] Dramatically improve Drupal-provided user interactions through perceived performance boosts and optimistic feedback
joachim’s picture

From this post https://www.drupal.org/node/2645250#comment-10774506:

> Also examining relatively quick wins like removing the full server-side render + http redirect we do for all non-AJAX form submissions, which already has an issue open.

That would presumably cut the time for a form submission by 50%. I don't know what the other issue reference by that is though.

catch’s picture

Found that issue. The big advantage of that is that for something like the comment form, it would allow us to dramatically improve performance while still serving zero javascript to anonymous users.

#2503429: [PP-*] Allow both AJAX and non-AJAX forms to POST to dedicated URLs

Wim Leers’s picture

From #80 in that issue:

RE: Also examining relatively quick wins like […]: I'd love to see what PJAX/Turbolinks would buy us in terms of making the UX faster: #2603794: Add PJAX/Turbolinks support. I think I'll be able to work on that somewhere in the next few weeks. It could of course end up being a marginal improvement. Let's find out.

david_garcia’s picture

This is starting to make sense. I just found it unbelievable that the other issues sort of proposed a full rearchitecturing or mixing node.js with php (WTF!) when there are things that can be done now with ease to vastly improve user response times and perceived performance by using some more AJAX and other quick and cheap tricks.

Keeping things server side IS important. If you mess up choosing your server side technology, then failing to choose the right one is not *too* critical because you control your servers. Moving too much things to the client and not choosing the right tool might mean that from one day to another you will need to completely rewrite a whole application. I have seen this happen with the poor users of FLEX, applications using XML Islands, and some others...

Drupal *is* moving to a space different than that of brochure sites and quick one shot projects, and these have other requirements to make the big investments reliable in the very long term. Or why do you think that COBOL (or even FORTRAN) programers still make a *very* good living? Because many banking (and other tough industries) still have +15 y/o stuff running and they want to keep it like that.

I have also seen failing (or being held back) full client HTML5 based applications that need advanced features to run (local databases, p2p, sockets, streaming and video capturing, etc..) because these edge features are not homogeneous among browsers.

The current Drupal AJAX framework still has a lot to say if we let it speak without asking too much from us. The problem is that this AJAX framework is just a Drupal thing and *many* are looking forward to jump off the island - or off the boat depending how you look at it - and investing time into this is not fun.

corbacho’s picture

Keep in mind that mobile offline usage of websites will become really popular (if not a requirement) during 2016-2017, specially after Safari & IE releases ServiceWorker. Then, don't matter how fast the server responses are .. you still want a powerful client side framework to keep user expectations

attiks’s picture

#7 I might become popular, there's no time line yet for service worker, and even if we have service workers, it doesn't mean you can load whatever you want on mobile devices, most will still be limited in storage space and speed. But service workers will make it easier to do, the biggest problem is that there's no polyfill, so starting to implement it will possible be hard.

The initial loading is still very important, so the server response has to be equally fast.

corbacho’s picture

@attiks I agree, server response times are very important. Service Workers kicks in *after* the first page view, and they can be used in a "progressive enhancement" fashion.

This issue is about thinking about "user experience goals", as Marc framed it.

Do we want to offer a nice offline UI experience for Drupal 9 ? Then, there is no alternative other than thinking about client side logic. And it could frame the solutions/ conversations in a different manner.
Chrome and FF have them, and both IE and Safari have ServiceWorkers "under consideration", and community is optimistic about it.

There is even a Google movement about rethinking websites as App shells (where, *after* the first page load, the website behaves more like an mobile app, so there is no need for server requests, even if you refresh the page. At least for basic skeleton of the page. https://developers.google.com/web/updates/2015/11/app-shell?hl=en
There are simpler "recipes" https://serviceworke.rs to take advantage of them, but most of them, they require a big chunck of logic in the client side.

Maybe, we could store the navigation menu in localStorage... or even an admin could create an article without internet connection?

I wanted to highlight this, specially for long term view (thinking about Drupal 9).
Today, most browsers are evergreen, so serviceworkers could be ready to use in all browsers, as early as 2017

catch’s picture

Do we want to offer a nice offline UI experience for Drupal 9 ?

This needs to be defined. For example, what is a 'nice offline UI experience' for Drupal.org?

You mentioned offline article creation, but most node types on Drupal.org contain entity references (project, issue tags, Assigned etc.). That's not at all unusual for other sites.

So possibly you could have a local copy of a node form, but you'd not be able to complete all the fields or run validation on them. This isn't an issue between server-side vs. client-side, it's a problem with how dynamic our entity forms are, and how much Drupal itself depends on interactions between different entities.

For me the first step here would be making AJAX form submissions more robust. For example if the site is not accessible, inform the user (in a nice way, not like autocomplete failures now where you still get an alert()) and preserve the form contents with the ability to try again.

borisson_’s picture

I think the first thing we should do is getting in javascript tests (#2469713: Step 2: Create a JavaScriptTestBase using PhantomJs Driver/Binary).

That way we can document the new behavior by adding new tests when we make changes to the user interactions. Wherever this issue leads, there will be added untestable complexity without getting in that issue.

Wim Leers’s picture

For me the first step here would be making AJAX form submissions more robust. For example if the site is not accessible, inform the user (in a nice way, not like autocomplete failures now where you still get an alert()) and preserve the form contents with the ability to try again.

Yes, Drupal's network/server error handling is traditionally extremely poor. (Try using the "Issue tags" autocomplete on this very issue on a network connection with a >1 s latency while on a train. Submit the form while the autocomplete is still … autocompleting. You'll get nasty/bizarre/byzantine errors thrown at you.) Improving that would already be a significant help in improving the UX using our existing AJAX system.

You mentioned offline article creation, but most node types on Drupal.org contain entity references (project, issue tags, Assigned etc.). That's not at all unusual for other sites.

Indeed! For many of these things, the server (generalized: a central authority) is absolutely required for consistency/correctness. Many sites/web apps with a great front-end work around this problem at least partially by not validating, i.e. by making references "pseudo-optional" (not finding the right terminology). A good example is mentioning another user on Facebook or Twitter. In principle, we'd do that in Drupal with a reference to a User entity. But such a reference is either valid or invalid in Drupal. On Facebook and Twitter, you just write @the_correct_name_or_something_that_I_remember_but_that_may_be_wrong, which maximizes correctness thanks to an autocomplete, but you're absolutely allowed to make invalid user references.
It's that sort of lack of strictness that also helps to enable better UX. But this is not acceptable for all use cases. And Drupal can be used for many use cases.

Interesting challenges ahead :)

catch’s picture

A good example is mentioning another user on Facebook or Twitter. In principle, we'd do that in Drupal with a reference to a User entity. But such a reference is either valid or invalid in Drupal. On Facebook and Twitter, you just write @the_correct_name_or_something_that_I_remember_but_that_may_be_wrong, which maximizes correctness thanks to an autocomplete, but you're absolutely allowed to make invalid user references.

So you could do this easily enough with:
- a filter that looks for @username
- js that autocompletes as you type @username
- a hidden field that stores the valid user-references found by the filter, if you needed that available in a structured way.

Somewhere, there's a Drupal.org issue asking for exactly this sort of feature for the issue queue, it's also similar to what Jira does. But as you point out, that's fine for some cases, not at all for others (i.e. the 'assigned' field requires a valid user in both Drupal and Jira, so the conflicting UX requirements co-exist in both forms).

attiks’s picture

#13 Unless you have as many users as this site, then you need to get back to the server, and it will be even more complicated if your users are LDAP users.

But as said in #12

Interesting challenges ahead :)
catch’s picture

@attiks well if it's just a mention like this one I started the comment with, the autocomplete could simply not fire at all in that case (or try and fail gracefully) - i.e. the same way you can compose a twitter post offline on a mobile app and the server will figure out it's a mention (or not) when it gets there. But yes it's still significantly degraded if you're offline.

attiks’s picture

#15 It wasn't meant as criticism, I just wanted to clarify it will all depend on the use case, sorry for the confusion.

catch’s picture

@attiks well no it's a good point - even the most optimized case we can think of for offline would still want autocomplete and hence server requests by default.

I tried to think of a counter-example of a form that could be safely filled 100% client-side, but it requires limiting to something like a specific survey form with no entity references at all. The contact form (assuming you don't add any fields to it) would be OK I guess. Offline submission of contact forms is really a very, very limited use-case though.

joachim’s picture

I've had a slightly wacky idea which I'm not sure is viable or not: what if we anticipate form dialogs and cache then pre-rendered?

For example:

1. user goes to edit a view
2. Drupal renders the main view form and delivers that, along with a token. In the shutdown for that page, it then goes on to render all the modal dialogs for that form, and cache them with the token.
3. User open a modal dialog, say, 'Add fields', or the edit dialog for an existing field
4. Drupal retrieves the rendered form for the modal from the cache and serves it

Where this all gets rather complicated is where the user's actions invalidate the cached modals: eg user adds a field, which invalidates the style settings; adds a relationship which invalidates the 'add field' dialog.

The other thing is that the user might make a modal dialog request while the server is still pre-rendering all the dialogs, as with Views there are rather a lot. The pre-rendering system would need to keep checking whether it's been superseded by a new request and should therefore give up.

tic2000’s picture

I'm trying to write a module to improve the back-end experience.

Some of the issues I found that will affect implementing improvements for front-end too.

1. D8 is smart enough to serve me an array of commands if I add the "use-ajax" class to a link. In theory this means I don't need to write code like in D7 with link using nojs/ajax. I thought I would use an alter hook to alter the response to do what I want. But in reality hook_ajax_render_alter() reminds me of hooks back in D6. I opened an issue for that #2655684: hook_ajax_render_alter() lacks context. That hook needs improvements. At least some route information, arguments passed to the route, the command used, etc.

2. Instead of loading a new page I load forms through AJAX and I attach those forms to the page. Now I want the link to just toggle the visibility of the form. But even if I remove the "use-ajax" class, the link still behaves the same and I can't easily detach the behavior and attach a new one. As a workaround I actually had two links on the page and remove the ajax one after the first click. For this there is already an issue open which should fix the detach problem. #2561619: Drupal Ajax objects and settings grows endlessly

attiks’s picture

Just thought of another improvement: why don't we replace the delete functionality with the following

- "Remove" button instead of "Delete"
- Clicking on it triggers an Ajax call and marks the entity as being removed
- User gets a message that the entity is removed
- User also gets an "Undo" link to reverse the action
- Housekeeping takes care of the actual deletes

Benefits:
- Instant feedback
- No need for confirmation screen
- A real way to solve the "Ooooo shit" situation

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.