Problem/Motivation

Due to evolution in the JavaScript framework landscape, many front-end developers today consider Backbone an obsolete framework for building user interfaces. Backbone is relatively unopinionated, lightweight, and flexible, but satisfying growing demands on state management is a considerable challenge. For instance, Backbone has no built-in data binding, which means that logic to update the view when the model changes must be written by the developer. As such, Backbone relies on getters and setters (similar to React) to detect changes and react to them. Finally, Backbone views effect changes directly in the DOM (like jQuery), detrimentally impacting view reusability (many frameworks today are exploring or using an abstract DOM approach such as vdom).

The purpose of this issue is to select a client-side framework (e.g. Angular, Ember, Knockout), library (e.g. React), or compile-to-JavaScript project (e.g. Elm, ClojureScript) that would supersede Backbone in UIs currently using it and aid the construction of other next-generation UIs. Such an adoption would prepare us for a potentially more JavaScript-driven future by attracting new developers accustomed to their framework's tools. It would also accelerate UX improvements by providing better code reusability across all of our UIs. Due to selective asset loading, this change would not break BC; any modules that wish to use the new framework can simply remove Backbone as a dependency and add the new canonical framework.

The scope of this issue is currently limited solely to site building and content administration UIs within the admin experience, such as Views UI, Field UI, and block placement. Nonetheless, UIs for authenticated users that lie outside the confines of admin themes, such as Quickedit, Toolbar, and Contextual Links, would also benefit from a new unified framework.

A framework-agnostic or API-first Drupal and choosing a new client-side framework for admin UIs are not mutually exclusive, as the latter will illuminate gaps in how we provision APIs. People who want to remain on a traditional little-or-no-JavaScript implementation should feel empowered to do so. Likewise, people who want to fully decouple and just use Drupal as a JSON API (the huge losses in Drupal functionality notwithstanding) should be able to do that too.

See #2651660: Investigate where and how a frontend framework could be used for emerging discussion on where and how a framework superseding Backbone would be used.

Proposed resolution

Key Drupal contributors have teamed up with experts on important client-side frameworks (including Angular 2, Ember, and Elm) to produce a comparison matrix to help determine what framework would be most appropriate for Drupal core. The matrix assesses Angular 2, Ember, React, Elm, and a few less promising frameworks: Backbone, Angular 1 and Knockout.

  • Are these the correct criteria with which to evaluate client-side frameworks to supersede Backbone (for progressive decoupling)?
  • Are there other frameworks that meet our minimum criteria?
  • Are the conclusions we’ve made about these criteria appropriate?

Remaining tasks

  • Agree on criteria for selecting a JavaScript framework to supersede Backbone in order to create a shortlist.
  • Develop proofs of concept that show the right framework for the job of controlling those UIs to be amended while still preserving key functionality (e.g. Drupal configuration, themes, and localization).
  • Determine when / in which Drupal version (8.1? 8.2? 9.0?) to add the framework. Note that core code can’t leverage the framework until the framework is in core.
  • Reach out to framework maintainers to resolve any considerations around licensing, release cadence, and inter-project cooperation.
  • Draft a change notice similar to that created for Backbone adoption.

User interface changes

Leveraging the chosen framework to improve core UIs would be separate follow-up issues.

API changes

None.

Data model changes

TBD, but likely none.

Further reading

Comments

prestonso created an issue. See original summary.

prestonso’s picture

Issue summary: View changes
droplet’s picture

Special thanks to the following experts who provided review and input:
Miško Hevery (creator of Angular; Google)
Igor Minar (technical lead for Angular; Google)
Ed Faulkner (core maintainer for Ember)

Must choose between these two. It seems other projects main developers are rejected / do not willing / no way to contact them to share their thoughts ??

Lesson Learned myself, one of them retired a popular project so quick.
https://github.com/facebookarchive/three20

the matrix missing vuejs, one of top 5 most starred in github I think.
https://github.com/vuejs/vue

fubhy’s picture

The matrix also misses PureScript, ClosureScript, Cycle, Riot and many more. For various reasons. I am sure that it is open for discussion though and not carved in stone. This is why this issue as created. To encourage further discussion, review and work on this topic.

effulgentsia’s picture

Must choose between these two.

No. One of the questions in the issue summary is explicitly "Are there other frameworks that meet our minimum criteria?".

It seems other projects main developers are rejected / do not willing / no way to contact them to share their thoughts ??

This is now a public issue, so we'd love other projects' experts to comment here. Those were just the ones we were able to reach and who had availability to work with us on the matrix in late December, during the holidays. The current state of that matrix is just a start though. This issue is about continuing it, in public.

Crell’s picture

Another requirement: No matter what Javascript-based fancying up we do, the page must still be usable without JS. That's been an expectation of Drupal for a long long time, even for admin pages, and it is one we should keep. Build an HTML/CSS-only admin page first, then add JS incrementally to make it whizz-bang. I am totally down for adding JS to make the admin more whiz-bang, but for many reasons the underlying "series of pages linked together" architecture that is the Web must be maintained, even for administrative pages.

nod_’s picture

Since this is an open discussion can we have more info about (from Dries' post):

… I believe that Drupal needs to move quickly to embrace a framework…

And here:

We need to accelerate the pace of these improvements to match users’ growing expectations.

What's the timeline Dries has in mind? Because this does not sounds like a 'it's ready when it's ready' deal and more of a 'it's ready when we make it ready'. If we want to timebox this whole discussion that's fine with me but it needs to be very explicit from the start.

( edit ) +1 to what Crell said. Though I see the "server rendering" justification coming :)

Crell’s picture

My other concern is that this is a very volatile part of the industry. We expect Drupal 8 core to live for several years; I expect we're 3-4 years away from seeing a 9.0 release, which means 8.x will be our "latest" for that whole time. However, Javascript frameworks at this point have a lifespan measured in months; a year is an old one. :-) Just as Backbone is now passe in the eyes of many, I expect every library listed above, including whichever one "wins", to be passe within 24 months at most. And we'll be right back here again. The client side moves almost dangerously fast right now. And that's not even taking into account the often ridiculous build toolchains needed.

I'm not much of a JS dev at this point so I have no particular favorite, but I worry about tying ourselves too closely to any of these tools.

Side note: Twig.js is a thing, although I don't know if it's still viable for us given how much we've done to extent server-side Twig (render arrays, etc.)

webchick’s picture

(responding to #8) I'm not as sure. When we chose jQuery, there were many other frameworks out there including Prototype, Mootools, Dojo, etc. We evaluated against some specific criteria—wanting something modular that would work well with Drupal, wanting something with a large, active community around it and a responsive maintainer—and that choice seems to have served us pretty well. I don't hear about any of those other frameworks anymore.

And even though the VCS landscape was far less volatile, at the time we selected Git there was at least one other very strong contender, which was Bazaar, development on which has since stalled. So obviously we made the right choice there, as well.

So to me, it's about picking the right criteria, talking a lot to people in the know, and gauging the community response. We've done pretty well with that, historically.

Crell’s picture

#9: To some extent, yes. However, to some extent Drupal's selection of jQuery helped it win (not entirely, sure, but it was a factor), and we were then always stuck on old versions of jQuery. :-) As for Git, there were other contenders but it seemed fairly clear that the "market" was trending toward Git in a hard way already. It wasn't anywhere near as volatile as the JS framework market is today.

As counterpoints, we picked Bootstrap (which we're now talking about replacing with something else) when it was a stable and mature leading tool. I don't think I'd say we "got that wrong", just that it went out of vogue long before jQuery did.

Also, we originally went with create.js for QuickEdit. We spent a lot of time trying to make that work, then had to pull out and roll our own atop CKEditor, which wasn't in a usable state when we initially started. There was a lot of flapping and churn in that process as we changed our minds about what tool we should be using. (I'm not saying we "got it wrong", more that the definition of "right" was a lot more fuzzy and shifting.)

I'm less concerned about us "picking wrong" than I am "none of these options will have the longevity we want". Even the popular darling Angular 1 is now dying already, replaced by, potentially, Angular 2 (different enough to share little but the name) and React. That doesn't mean Angular 1 would have been a "wrong" choice a year ago; it's just that the market is moving that quickly.

mdrummond’s picture

I'd really like to see the following clarified:

- I think we're talking about using a JS framework to render admin pages on the server and then handle post-render interactions?
- I'm assuming we're not talking having every user-facing theme use this JS framework to render content by default?
- What about UI components that appear for authenticated users on user-facing pages, like Toolbar, Quick Edit and Contextual Links. Would those use this JS framework?

We've had a lot of challenges with how to properly handle these front-end admin components. It's complicated, because generally you probably want those components to match the backend theme, but frontend themes might want to override some things with those components at times. I had an idea for how to improve that experience in #2632584: Add a "public admin" theme subtype so that admin themes can have subthemes for front-end UI components. Frankly if we end up using a JS framework to render the admin pages, and a more normal Drupal process for user-facing themes, I think this intersection of admin UI and user facing themes becomes even more complicated.

I'm trying to reserve judgment until the picture is clearer, but I'm pretty concerned about this.

We just finished a contributor-driven initiative to overhaul how people interact with Drupal's markup through Twig. We've worked very deliberately to make it easy for people to edit markup without having to know PHP. Now we're talking about potentially throwing out that work and mixing Drupal markup with JS, which is pretty much the same situation we just got out of. I know there are loads of people who love working with JS that think mixing the two has a ton of advantages. There are also loads that were excited about D8's front-end improvements. How is this going to work? Some sort of JS templates for the admin side, and a completely different set of Twig templates for user-facing pages? People now need to two completely different way to work with Drupal markup?

I'm also really bothered that this is a completely top-down approach. We are being told this is what Drupal is going to do moving forward. Nobody brought this up with the folks who did the bulk of the work on Drupal 8's front end in the last year or two (particularly with the Twig initiative), at least that I know. We're potentially radically changing Drupal's theme system, and I guess we get to make a few comments, then Dries decides what we are all supposed to do. Yes, he's the benevolent dictator, but this doesn't feel at all like a community driven process.

I think it's also worthwhile to keep in mind that changing Drupal's front-end templating from PHPTemplate to Twig took several years! It feels like this is being discussed like we are going to make a decision now based on the current state of the JS landscape, and then bam, Drupal takes advantage of that. In reality, depending on how much of Drupal's theme system is going to use these JS frameworks, this could be a hugely complicated and time-consuming process. It's certainly not going to take weeks, and I doubt even a few months with loads of people working hard on this would do the trick. We're talking about a year or two or three, in all likelihood, in which time the landscape may shift entirely again.

effulgentsia’s picture

No matter what Javascript-based fancying up we do, the page must still be usable without JS.

If by "JS" in that quote you mean client-side JS, then +1. I agree with the fundamental principle that an open web includes being open to people who wish to disable browser JS for whatever reason they have to do so. Semi-related: http://thewebahead.net/110 is worth a listen.

I see the "server rendering" justification coming

Yep, good foresight: I propose #2645666: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS for Drupal 9. In terms of how it impacts this 8.x issue, personally, I'd be ok with any "next-generation admin UIs" built in 8.x to require Node.js, so long as sites without Node.js can continue using the non-next-generation UIs. That's just my opinion though, I'm sure it'll get debated. Or if a JS framework/architecture is chosen that works with PHP-generated HTML as an effective fallback for when browser JS is disabled, that's fine too.

The purpose of this issue is to select a client-side framework (e.g. Angular, Ember, Knockout), library (e.g. React), or compile-to-JavaScript project (e.g. Elm, ClojureScript) that would help us build next-generation admin UIs.

I think considering admin UIs is certainly necessary. However, I think it's also important to consider site visitor facing UIs too. And as a lead up to responding to @mdrummond's points in #11, I'd like to offer a motivation for why looking into these JS frameworks is so important. Consider the process of commenting on a piece of site content. Currently in Drupal, there's latency between clicking Reply and getting a reply form, and latency between typing a comment, clicking Preview, and seeing a Preview, and latency between clicking Save and seeing the comment appear on the original page. Even if we replace the full page refreshes with a Drupal Ajax UI that only replaces the DOM element of the commenting area, we'd still have those latencies while waiting for the server to render each of those steps. And those latencies is the difference between a user feeling like they are doing something and them feeling like they are ordering a computer to do something and waiting for the computer to do it. Which changes the entire feel of the process. And I would argue that now that a billion people have experience with commenting in latency-free Facebook, that Drupal's with-latency commenting process feels weird and makes you wonder if it's even reliable, much like what happens with the aging of any technology (e.g., visual effects in movies). And the reason I bring this up is that if open-source software like Drupal doesn't provide what will soon be considered a minimum acceptable commenting experience, then people will be tempted to move the conversation to services like Facebook or Disqus. But in my opinion, that's a really bad result, especially for conversations about sensitive topics (say, a support group for people recovering from abuse) whose content shouldn't be in the hands of those companies. Having said that, I'm ok if this issue focuses more on admin UIs as an interim step. But per #2645666: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS, I think eventually, all Drupal UIs should be capable of latency-free client-side re-rendering in response to client-side interaction.

I'm also really bothered that this is a completely top-down approach. We are being told this is what Drupal is going to do moving forward. Nobody brought this up with the folks who did the bulk of the work on Drupal 8's front end in the last year or two (particularly with the Twig initiative), at least that I know. We're potentially radically changing Drupal's theme system, and I guess we get to make a few comments, then Dries decides what we are all supposed to do. Yes, he's the benevolent dictator, but this doesn't feel at all like a community driven process.

I don't think that's what's going on at all. As Drupal's project lead, Dries has been bringing up in multiple blog posts and keynotes the topic of decoupling, with last month's blog post being perhaps his strongest statement about this being a really important topic for Drupal's future. But he's outlining a broad vision:

  • That latency-free UIs are increasingly important for Drupal to embrace and support.
  • That using a JS framework/library that's more capable than Backbone (which doesn't do any client-side rendering at all on its own: in Drupal core's case, we hand that off to Underscore, but Underscore itself and our usage of it is very bare-bones) is likely to be necessary.
  • That picking one to include in core is necessary in order to use it for core UIs and a helpful default option for contrib to standardize on.
  • And that he'd rather we not postpone all this on a clear market winner emerging, because that could take years and he wants us improving Drupal UIs before then.

As to whether we pick Twig.js for client-side rendering and stick with Backbone for state management, or whether we pick Ember and move templating to Handlebars (which is very similar to Twig), or whether we pick Ember and try to make it work with Twig templates, or whether we pick React and move templating to JSX, or ..., he's leaving that to a community process. Hence this issue. The current matrix is just a start of the conversation, and it includes a row for "Template engine friendliness for Drupal themers", and as part of this issue, we can improve that row or add more rows.

We've worked very deliberately to make it easy for people to edit markup without having to know PHP. Now we're talking about potentially throwing out that work and mixing Drupal markup with JS, which is pretty much the same situation we just got out of.

Well, if the solution that best meets all the needs that we need to consider (not solely theming needs) ends up being to use Twig.js or to change the templates to Handlebars, then we won't have that problem. But if the best solution is to use something like React+JSX or Elm+ElmHtml, then you're right about the concern. However, I don't think it's the same situation. With markup in tpl.php files, we had the problems of ugly PHP syntax for simple operations, no opportunity to auto-sanitize printed strings, leaking string vs. array vs. object implementation details into the templates, and no way to prevent templates from invoking dangerous server/database operations. JSX and ElmHtml might not be subject to any of those problems (I don't know for sure, we'd need to evaluate further). As to whether there are other problems with JSX / ElmHtml, I think we'd need to do some more exploration to properly evaluate.

People now need to two completely different way to work with Drupal markup?

Yeah, eventually I think we'll need to bring it back to one way, which I propose in #2645666: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS to be whatever way is best for JS (which may or may not be JSX / ElmHtml). But how to manage two template sets in 8.x (if it comes to that) is a very good question. If nothing else, that's a huge point in favor of finding a JS solution that can work with Twig templates. But if we can't find such a solution (e.g., if React+JSX offers too many compelling benefits to pass up), then let's explore options for how to make the transition state of 8.x as sane as possible.

giorgio79’s picture

+1 for the NodeJS route: https://www.drupal.org/project/nodejs

Here is a video for Private Message integration https://www.youtube.com/watch?v=f8XAT1oB1eA&ab_channel=ASYNCDrupal. NodeJS already has Views and other integrations too...

Dries’s picture

I personally think we can have different rules for the front-facing site experience (for site visitors or end users) and for the administration experience (for content editors and site builders). I agree that we can't require site visitors to require JavaScript, but I do believe it would be reasonable to require content editors and site builders to have JavaScript enabled.

David_Rothstein’s picture

it would be reasonable to require content editors and site builders to have JavaScript enabled

^^^ This would mean JavaScript is required to contribute on drupal.org (or other community-based Drupal sites). I don't know if people think that's a bad thing but it's worth pointing out.

On the overall issue, wouldn't it benefit from first having a specific design/mockup/prototype for exactly what Drupal's user interface (or at least an example part of it) should look like in the future, and then pick a framework afterwards? The Calypso writeup is a good read in that regard, although it defers a discussion of exactly why they chose React to a future post.

David_Rothstein’s picture

Issue tags: +needs backport to D7

It's at least worth considering adding this to all supported versions of Drupal core.

pbuyle’s picture

On his blog, Campbell Vertesi made the interesting point that contrib is a great place to handle the ever-changing, rapid landscape of front end frameworks. All Drupal core has to do is make it easy.

IMHO, Drupal core needs to standardize on a good font-end DI container, (polyfills for) standard language features (eg. ES6 module loading) and APIs (eg. Fetch API for HTTP request) and lightweight decoupled front-end components. So any front-end frameworks or libraries can easily re-use these lightweight decoupled components and standard APIs. all they need to do is being able to interface with the Drupal's front-end DI.

Once this is done, using a components based approach, Drupal core could then standardize on a view library for its own UI. Having a clean separation between the components (and the DI) and the view library would allow Drupal core to adopt and switch the later more easily. It would also allow the administration backend to move at a different speed than the visitor front-end. Finally, it will minimize overhead should two view libraries be used during a transition period. Better having React and Polymer loaded on the same page, than loading Ember and Angular.

If the framework approach is chosen, the ability to replace its components easily and being non-opinionated on the applications architecture and dataflow are determining factors. Sadly, most font-end framework are not designed this way. Backbone was, but it lost is momentum long ago.

TL;DR: Consider not using a framework but a combo of interchangeable front-end library/tool. Consider Polymer.

Note: In the choice of jQuery, front-end developers are still annoyed of by how difficult this is to change the jQuery version for the front-end. So I would not call jQuery integration in Drupal a total success.

Crell’s picture

One of the many advantages of a progressive-enhancement approach: If core provides a solid "traditional" approach with the right hooks, then contrib-based admin themes can experiment, evolve, and Darwin it out while still providing that more integrated, Facebook-esque experience. Distros like Lightning or Panolopy can easily pick up one or another. That makes it really easy in 8.4 or something to decide if one has clearly "won", or been done best, or whatever, and just toss that into core beside/instead of Seven. And (and this is super important) we keep it sufficiently separate that it's easy to swap back to Seven, or the theme itself degrades gracefully, if on a given site you don't want to go that far. We keep the traditional approach working, ever and always, which gives us a stronger foundation to build the glitz on top of anyway.

That is a much lower risk approach for us to take than saying "rewrite the whole admin in framework X, even though it's only 3 months old".

rachel_norfolk’s picture

Given one of my personal bugbears at the moment is staring at a blank screen on my phone waiting for js-heavy pages to download from certain websites when I’m out in the sticks, the very last thing I would want to be stuck with is needing to do a content update on a site whilst out and about and not being able to.

I get the idea of a more comprehensive framework for handling some of the complex interfaces we have for, say, views, but such a large proportion of admin pages are and should be simple. We’re getting really quite fab now at sending ‘good’ html forms for admin pages that require the absolute minimum from our connections. I would hate to see us lose that.

Is there an “opt-in” scenario we could come up with, like Crell suggested at #18 with something initially in contrib that does “advanced admin” that might move into core later? (and still be optionally enabled)

Jose Reyero’s picture

First of all I strongly disagree with the Problem/Motivation part of the issue. I don't see the problem, nor anything there that cannot be done with current AJAX framework, maybe some improvements to that would be fine but that's it.

Frameworks are always nice, but including them into Drupal core means basically "enforcing them". Once enough contrib modules have used the framework, they just will do if it is in core, site builders won't have other option than using it... then using any other framework for the site itself will mean loading two frameworks... Then we are at the starting point again.

In other words, if we let a fancy js framework in Drupal core, that will become the one and only framework for Drupal sites. And this is a serious vendor lock-in, since all these frameworks are pretty new and constantly evolving.

So my option for the client-side framework in core would be: None.

Putting it some other way, front-end frameworks should be pluggable and disposable. And they need to be in Contrib for that.

I also agree with Crell #18, though I'd suggest a bit different way forward: Add more reusable widgets / components in Drupal core and use them. Then any contrib module can just rewrite those widgets using any nice framework, or none...

yukare’s picture

So we have to learn twig now just to drop it on next Drupal? And drop php for javascript while in this ?

bojanz’s picture

Frameworks are always nice, but including them into Drupal core means basically "enforcing them". Once enough contrib modules have used the framework, they just will do if it is in core, site builders won't have other option than using it... then using any other framework for the site itself will mean loading two frameworks... Then we are at the starting point again.

In other words, if we let a fancy js framework in Drupal core, that will become the one and only framework for Drupal sites. And this is a serious vendor lock-in, since all these frameworks are pretty new and constantly evolving.

So my option for the client-side framework in core would be: None.

Putting it some other way, front-end frameworks should be pluggable and disposable. And they need to be in Contrib for that.

So what happens when Media requires Ember and Commerce requires React? Two mega frameworks, and one very heavy page.
The client-side framework selection will never happen in contrib, for fear of conflict, and for fear of losing convenience ("To install Media you must first install the Ember module").

On the other hand, I do want to point out that Drupal is a CMS, serving content. Outside of the admin panel, few pages have a need for fancy JS. Commerce has quite a few (cart, checkout, etc), but that doesn't qualify as an ordinary page (as represented by a node or a view).
This makes a component-based approach necessary, an all-or-nothing framework won't do it for us.

Jose Reyero’s picture

@bojanz,

So what happens when Media requires Ember and Commerce requires React? Two mega frameworks, and one very heavy page.

I'd call it a bug. Site building modules or API modules should not enforce end-user UI frameworks. That may be different for "Distributions" though, like 'Commerce Kickstart' that is more a site solution.

The client-side framework selection will never happen in contrib, for fear of conflict,

That's my hope, most modules will just won't require those frameworks, will manage without them, I will be able to use them for my site solution with whatever other framework I decide.

Outside of the admin panel, few pages have a need for fancy JS.

I think it will be quite the opposite. And worse once every contrib module starts providing nicer widgets because they just get the 'fancy js' for free.

But... do we at least agree that once we get a framework into core, that will be the one and only option for Drupal sites?

bojanz’s picture

@Jose Reyero
Thanks for clarifying, didn't realize your solution was "do nothing, don't address the problem".

Jose Reyero’s picture

@bojanz,
Nope, "my solution" is: "Do nothing in core, don't make the problem worse".

Just because I believe that adding the framework in core is just an expensive workaround.

attiks’s picture

My 2 cents

I feel the way Problem/Motivation is worded, it sounds like the only solution is to add a framework to core. After reading all comments advocating to do so, I'm not convinced it is really needed.

The problems we're facing when building Drupal sites (most of what follows applies to D7) are mostly not related to the absence of such a framework. Depending on the site we use one of the following strategies to make it snappier in the front-end:

  1. Use core/contrib javascript with or without an updated jQuery
  2. Use core/contrib javascript with a lightweight alternative (like Zepto)
  3. Ditch all core/contrib javascript and use vanilla javascript

For the projects were we customize the back-end entirely, we stick with jQuery, but we improve the PHP side especially for slow pages, like for instance the permissions page. Dries' posts and the discussion that started afterwards on Twitter made me release some code on how this could be done, feel free to have a look at Dream permissions project

I think we as a community can better focus on how we can make it easy for core/contrib to start using fancy UI elements, https://www.polymer-project.org/1.0/ comes to mind, if we can provide an easy way (like render elements) for core/contrib to use them, it will make everybody happy.

To conclude, yes we have to do something, but as pointed out by @Crell, @mdrummond, @rachel_norfolk, @bojanz, @Jose Reyero and others, the solution is not adding a framework, progressive enhancement/Cut the mustard is to way to go and not only for Drupal.

Regarding @effulgentsia comment #12: Comments is kind of a special beast, most of it can be solved by using plain javascript if you render the reply box on the same page as the node. It is also a bit misleading both facebook and disqus works slowly on mobile and both of them needs extra request to open/start the comment box, which leads to mobile phones having to activate the radio and start fetching resources. Tested on https://m.facebook.com: 15 requests, 120KB, 2+ seconds

Crell’s picture

Whilst it's easy for me to say this as a non-frontend-dev, for situations like the Comment form, what in the status quo is preventing us from making it more whiz-bang with just modern-JS and jQuery as needed? What in the existing Ajax framework would make a "press the button, new comment appears immediately, comment is posted async and we assume it will save" type module really hard to do?

Off hand (again, speaking as someone who knows only enough JS to be dangerous), the only part that sounds even slightly hard to me is the "render new comment immediately before Drupal responds" part. The rest should be fairly dead-simple now with the improvements that have been made to core. And that part is mainly a double-theming problem (server-side and client-side), which won't be solved by adding Angular, React, Ember, or whatever else to Drupal. We'd still need a template in both places.

So... someone help me out here. What's the pain point that we have currently that a standard large JS framework would solve, that smart use of the Ajax API, jQuery, and Javascript that relies on a browser written after the Bush presidency ended couldn't?

catch’s picture

@attiksThe comment example is a good one to unpick things. For example there's text formats, image and other file uploads to deal with. All of those will be very difficult to handle client side in terms of instant rendering.

Another way to approach that would be draft + autosave support, then actually posting the comment is just switching the published status and it'll already be available otherwise for preview. Same goes for previews of other entities as well. All of which as you say can be done with Ajax.

attiks’s picture

#27/#28 The most difficult to implement would be a generic solution that will work for all possible comment forms, since these are fieldable and nestable, you don't know what you have on the client side. Since they are extendable as well using PHP (hooks) you have no idea what's going on on the server side, things like auto generate title, mark a user depending on his/her role.

So the only proper solution is to have a framework that supports all of the above, including a magic translation of all PHP logic to javascript. If we as a community want this I see 2 viable solutions, switch to javascript server side or add support for PHP client side, but this is being discussed in #2645666: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS. For the moment we handle this on a project per project basis, if you know how the form will look and what logic is added on the PHP side, you can build a snappy comment form (even in D7).

You cannot compare this to commenting on FB, they have one specific form to deal with.

corbacho’s picture

Sounds too risky to bet on a big strongly opinionated JS framework at this point. I dislike Ember, just because it's the biggest one. We are killing the mobile UX just from the start.

It's a pity BackboneJS has lost momentum, it was a perfect progressive step-up for us: easy to learn, small, compatible with jQuery. But the world has moved on, Backbone and Coffescript have got stuck in the ES5 era.

It's not only about choosing a framework for its merits, it's the baggage the bring with them. React, Angular and Ember are very opinionated, and bring their own "tooling pack" that alienates communities.

You can't swap them so it makes it look scary to bet on a single framework.

At this point, If I would have to consider a framework, I would like VueJS or RiotJS to be at least considered. The documentation of VueJS about how it compares to React/Angular/Ember/Polymer is a must-read: http://vuejs.org/guide/comparison.html#React

Vue is smaller (~24kb min+gzip) supports IE9 and newer. No jQuery dependency. "Provide the benefits of reactive data binding and composable view components with an API that is as simple as possible". MIT licensed. It's not the most popular at this point, but during 2016 I'm sure it will "catch up" with rest.

Choosing Vuejs, ironically, sounds like a safer bet to me, at this point, than other bigger, more popular frameworks.

I don't mind that it's not isomorphic. It's overrated concept IMO, and I would bet on PHP to level-up soon (as Crell pointed) during the next years, instead of adding a nodejs dependency.

Also, see author words about VueJS in 2015. Highlights on being adopted by Laravel community,.. and VueJS being a progressive framework:
http://blog.evanyou.me/2015/12/20/vuejs-2015-in-review

For reference, two surveys to JS developers were published recently. We can't overlook that jQuery still is the number 1 library and the most familiar for every developer
http://ashleynolan.co.uk/blog/frontend-tooling-survey-2015-results
https://ponyfoo.com/articles/javascript-developer-survey-results

mrf’s picture

I feel like we have jumped straight to "which JS framework should be in core" skipping right past the discussion of whether or not it is a good idea for us to rely on a JS framework in core, and how much we want to rely on it on that framework outside of the administration interface.

I would be much more comfortable if this issue focused on "Backbone is obsolete, what did Backbone solve for us, and what do we replace it with?" and we stayed away from the grandiose vision of one particular framework solving all of our current and future front end problems.

Finding a replacement for Backbone, or rewriting that functionality without a framework should be the #1 JS in core problem to be solved in the current cycle. I don't think anyone on either side of this issue would argue that we should keep Backbone.

Part of this discussion should dig into history and figure out how we ended up picking the wrong horse when it came to Backbone and how we avoid that in our next decision especially considering the quick turnover and short lifespan of JS frameworks.

chx’s picture

I was doing a little research about longevity and stumbled on http://blog.bitovi.com/longevity-or-lack-thereof-in-javascript-frameworks/ which is obviously biased but it is still an excellent read and needs consideration. Note this is not a vote for the framework they propose merely an endorsement of adjusting our evaluation criteria.

techlust’s picture

Longevity should be a factor but I have to question people arguing that Angular, React & Ember won't be around in a few years.

The enthusiastic (much more so than YUI etc) corporate backing and adoption (https://github.com/facebook/react/wiki/Sites-Using-React) has to count for something. It's hard to imagine Wordpress, Netflix or Facebook completely dropping React over the next 5 years when it's already being used in so many critical applications.

mrf’s picture

#1149866: Add Backbone.js and Underscore.js to core seems to be the best place to look at how and why Backbone was included in core and the discussion around that decision.

WIthin that issue there are references to both an improved authoring experience and onslaught of new front-end developers are both referenced as motivation for its inclusion.

We did get an improved authoring experience, but the onslaught of new front end devs did not occur.

It feels like we are making the same arguments for framework-next as we were for framework-last. I am wondering if the choice of framework is a red-herring for the continued lack of front end devs involved in core?

xjm’s picture

I just wanted to point out for @mdrummond and others with concerns about how this affects the theme layer that minor versions of Drupal (so Drupal 8.1.x, 8.2.x, etc.) can only add new APIs and functionality, and must retain backward compatibility. So breaking the render and theme APIs is not on the table for 8.1.x or any minor, and to me, that would include not breaking support for Twig. For example, the Stable theme is supposed to be, well, Stable. :) This was also my foremost concern originally upon hearing this discussion.

So for me the big picture is more about making Drupal friendlier for JS frameworks/frontends in general, and exploring how we could iterate that with API additions and optional new functionality. Picking a specific framework is a decision that would merely affect how much mileage we get out of these additions -- thence this issue, I believe.

Finally, developer experience, themer experience, testability, and security are all important criteria. Edit: For both the potential framework and the architecture we might use with it.

catch’s picture

#31 is a good comment - i.e. we're using backbone for a number of core features, it needs to be replaced, and then we might want to update the Views, Field UI, Permissions and other large/complex interfaces to use whatever the replacement is during the 8.1.x cycle. This issue then becomes a lot more straightforward if you think like that.

and how much we want to rely on it on that framework outside of the administration interface.

So in general, we aim to serve anonymous page requests without any JavaScript at all by default, and I think we should keep that aim. Or if we don't there needs to be a very, very clear trade-off.

Where this gets tricky is entity forms - at least node, comment and user accounts are exposed to authenticated users and to an extent anonymous users very easily in the standard profile, and the comment form is on regular pages (such as Drupal.org) where performance is extremely important. Views exposed filters and similar are another case.

So I think it would be worth trying to restrict the discussion to choosing a new client side framework for site building and content administration interfaces. For now, assume that core at least won't use it for anything visitor-facing, which means not on content entity forms and some others.

That doesn't mean never updating the comment form or user account form etc., but doing so we'd start a new discussion for (and I don't think it would start with a discussion about a framework in those cases).

Wim Leers’s picture

Title: [META] Select a new client-side framework for Drupal core » [META] Select a new client-side framework for Drupal core's admin UIs

For now, assume that core at least won't use it for anything visitor-facing, which means not on content entity forms and some others.

+1. This also means all the great new front end things in Drupal 8 remain totally unaffected (from all things Twig to improved asset handling to …).

And in fact… this is exactly what that the IS already proposes. The first sentence reads: The fluid experience in well-written single-page applications today far exceeds what Drupal’s admin layer provides. And further down the issue summary, it keeps stating "admin UI". @Dries repeated this again in #14.

So, let's make that more explicit by updating the issue title.

(I suspect if this were the title in the first place, this would've helped avoid a lot of drama.)

nod_’s picture

Title: [META] Select a new client-side framework for Drupal core's admin UIs » [META] Select a new client-side framework for Drupal core

Dries reasons for this are to improve UX and get new people. I tried to come up with deeper reasons that would make be back this up but it turns out it was just wishful thinking. Improving UX has nothing to do with a framework and everything to do with our issue workflow. It's never the code that really holds up UX issues. Now imagine anyone else other than Dries pushing this issue with such weak arguments, it would not have gotten on anyone's radar.

Proposing such disrupting changes without a strong case would hardly get past core committers and this is essentially what we're being asked to overlook now. There are a few claims in the IS that don't sit too well with me either:

  • "For example, it is no longer acceptable to experience a full page refresh when deleting a node or applying filters to searches of content." Says which study?
  • "It is also extremely important that when performing any action, users get immediate feedback (or even optimistic feedback, in which the user sees the result even before getting a response from the server from their action, e.g. liking a Facebook post triggers an instantaneous response)." I do agree with it but same thing, any study to back that up? surely not all interactions get the same benefit out of having instant responses, might even be harmful in some cases, who know?
  • "Application-like UIs in Drupal (e.g. Views UI and block placement) would benefit from a standard framework to enable a fluid experience." this is on the verge of marketing talk. If it means consider views UI as a single page app, block placement as a single page app it would be more explicit.

Attracting contributors has nothing to do with the framework we'd eventually choose:

  1. Drupal is a CMS, that limits the number of people interested in the problem space to begin with
  2. We're not on github and there is no pull requests, pretty much all (if not all) of npm is on github that's what the "mass" of js devs are used to
  3. Getting in the Drupal queue from the outside takes some serious will

What we have going for us is that we're using eslint and all our code validates, that's about it. Adding a framework will reduce the number of people from #1 but would slightly increase the motivation of #2. Given our strict (and fatally tedious) core process we'd be lucky to get even one person to stick to the community. Maybe there would be more js devs on Drupal projects, but with all the barriers to contributing I'm pretty sure they would say "Drupal sucks" and go around whatever issue they have. Ignoring the fact that a good JS dev is harder to find than a good Drupal dev.

To be honest in the issue summary "Adopting a framework with a better front-end developer experience would also attract new front-end contributors to Drupal." needs to be removed from the IS summary, that's not true. Still waiting for the swarm of new Backbone contributors.

Given all that we are left with three solutions

  1. Do nothing, arguments are too weak and the solution doesn't even address exposed issues.
  2. Select a framework get it in core for contrib to use and do nothing with it. Wait until panels or something else make use of it before starting to port core over.
  3. Get some real arguments behind the proposal and get the process back on track.
nod_’s picture

Title: [META] Select a new client-side framework for Drupal core » [META] Select a new client-side framework for Drupal core's admin UIs
catch’s picture

There are a few claims in the IS that don't sit too well with me either:

"For example, it is no longer acceptable to experience a full page refresh when deleting a node or applying filters to searches of content." Says which study?
"It is also extremely important that when performing any action, users get immediate feedback (or even optimistic feedback, in which the user sees the result even before getting a response from the server from their action, e.g. liking a Facebook post triggers an instantaneous response)." I do agree with it but same thing, any study to back that up? surely not all interactions get the same benefit out of having instant responses, might even be harmful in some cases, who know?

More than that, even if we accept that these is a problem, if we're only going to use a framework for site building/content editing UIs, then there's no way it's going to affect either of these cases for site visitors anyway. So we can open new feature requests for these (AJAX-enabled entity forms in core, instant feedback for AJAX operations), but neither should be in the issue summary for this issue.

Re-title looks like a cross-post so switching back.

swentel’s picture

Maybe we should ask feedback from the Panels maintainers. They've chosen to use backbone for Panels IPE for the Drupal 8 version, and as far is I know the first contrib to use it (there might be others, don't know about that). What's their experience, pro's and cons why they made the decision etc ?

nod_’s picture

Acquia lift uses backbone too.

rachel_norfolk’s picture

I have to hugely agree with nod_ at #38 that it really does feel we’re missing the point here. If this is about attracting frontend developers, adding a framework will not achieve that in any way I can think of.

Another query strikes me, too. Presumably, the thinking above was to replace Backbone with something? If so, isn’t that an API change of sorts? People will no longer be able to assume Backbone is available and current js api calls will fail. That means this needs to be a 9.x issue, not an 8.1.x issue?

almaudoh’s picture

Interesting point I've observed: most (if not all) of the people pushing for this so far are PHP devs; all the front-end people in this thread are pushing back. That tells a whole story :).
We definitely need more front end guys commenting here.

fago’s picture

I can totally see the merits in selecting a client-side framework, so it's easy to go fully decoupled if you are doing a project where it makes sense.

However, I'm not sure "progressive decoupling" is the right goal - doesn't that come with a lot of added complexity? It means the front-end rendering process always ends up being partly server-side in PHP/twig and partly on the client side. So wouldn't we end up with two different, rather complex systems that have to be properly integrated to work properly together? so once you have troubles, you have to debug across two different systems...?

Instead, wouldn't it be easier to vastly improve the Drupal ajax system and keep all rendering server-side, e.g. to natively support history updates, improve ajax performance, e.g. add some optimistic feedback etc. Couldn't we achieve the goals this way while keeping all rendering server-side?
I see that optimistic feedback is probably the hardest part here. It should be pre-pare and send simple feedback together with the initial page load, but optimistic feedback that involves user input would be obviously harder. So we'd have to clarify first, what kind of optimistic feedback we'd like to do.

fago’s picture

So I basically second the questions crell raised in #27, what are the points an improved ajax framework couldn't handle?

catch has a good point in #28:

Another way to approach that would be draft + autosave support, then actually posting the comment is just switching the published status and it'll already be available otherwise for preview. Same goes for previews of other entities as well. All of which as you say can be done with Ajax.

Thus, optimistic feedback involving user input wouldn't be possible with a client-side framework either as long as the whole input filtering pipeline isn't re-implemented in javascript. (What sounds a lot like moving towards node.js in general).

Wim Leers’s picture

Using the AJAX framework doesn't get rid of latency: you always need a round trip. That's the biggest problem. Even in an imaginary world where the AJAX framework communicates via websockets (which would surely reduce the server latency i.e. by avoiding a Drupal bootstrap), it still cannot remove network latency.

That's the key thing: to eliminate network latency, you must run the logic on the client side, which requires JavaScript.

(Not saying whether that's desirable, just observing the facts.)


Agreed on optimistic feedback involving user feedback being impossible without duplicating/rewriting boatloads of things in JS. Which is exactly what I said in #2645666-3: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS.

catch’s picture

Using the AJAX framework doesn't get rid of latency: you always need a round trip. That's the biggest problem. Even in an imaginary world where the AJAX framework communicates via websockets (which would surely reduce the server latency i.e. by avoiding a Drupal bootstrap), it still cannot remove network latency.

If you're doing entity previews, updating Views, inline editing etc., then in most cases we want to do a network round trip to ensure the current state of the object being worked on is in the database - whether that's an entity revision, temp store, key/value etc.

Even if we were to figure out client-side rendering of changes to Views forms (which seems extremely tricky - if you select a plugin, you need to get the options for that plugin, which is going to be a round-trip, or you have to pre-load them, which still has to come from the server beforehand), then I don't think we'd drop writing that change back to temp store. The main difference is it wouldn't be blocking and could happen in the background - but that backgrounding feels solveable in at least some cases while still relying on server-side rendering.

Wim Leers’s picture

Good points! Especially ensure the current state of the object being worked on is in the database.

The main difference is it wouldn't be blocking and could happen in the background - but that backgrounding feels solveable in at least some cases while still relying on server-side rendering.

I've also thought about this. The difference is that doing the logic on the client-side makes it simple. No shortcuts necessary. Optimistic feedback without having the logic on the client means taking shortcuts while doing the preview, and possibly replacing that instantaneous-but-low-fidelity preview with a high-fidelity preview upon getting a response from the server.

It'll be interesting to see what we can come up with. (I'm particularly thinking of the filter system, for example when posting/previewing a comment.)

fgm’s picture

Do we really want to go that route ? Insightful (mostly) on-topic about JS frameworks : https://medium.com/@wob/the-sad-state-of-web-development-1603a861d29f#.x...

effulgentsia’s picture

People now need to two completely different way to work with Drupal markup?

That is the situation we have right now. For example, this line for adding an "Add" button in Views UI and this whole file of QuickEdit components. In other words, we already have UIs where we're not using a Drupal AJAX request and server-side rendering, because the latency of that would be silly.

Maybe we should ask feedback from the Panels maintainers. They've chosen to use backbone for Panels IPE for the Drupal 8 version, and as far is I know the first contrib to use it (there might be others, don't know about that). What's their experience, pro's and cons why they made the decision etc ?

I'm not a Panels maintainer, so I can't answer that, but just a quick look at the code shows that they're continuing core's pattern of inlining client-side HTML into the .js files, with the addition of also using some Underscore.js template syntax.

Do nothing, arguments are too weak and the solution doesn't even address exposed issues.

Answering this issue with "do nothing" would probably result in:

  • Some contrib modules (like Panels IPE) getting ported with the Backbone/Underscore pattern, which requires themes to use completely different patterns to customize that markup (it's not Twig, do themers even know what the API is to swap out Backbone views and Underscore templates, there are no preprocess functions, etc.).
  • Other contrib module maintainers deciding that Backbone/Underscore is too limited and therefore picking Ember or Angular or React or Elm, each one with its own different approach to client-side markup generation. Then if a theme for a particular site wants to customize the markup across all these different contrib modules, it will need to learn the way to do so for all of these frameworks.
  • If in Drupal 8.2 or 8.3, we want to add any of these modules to core (e.g., Panels or Media or ?), then if what these modules ended up picking is not the framework we want in core, then a rewrite to the framework we do want in core will be needed.

Maybe the above is an ok situation. Maybe we need the additional exploration time of each contrib module picking its own approach with no nudging of any kind from core, and then we look at the resulting mess in a year or two and figure out how to align everything then. But I'm not yet convinced that that's the best option. I think it ought to be possible for us to explore these frameworks a bit more, and based on that exploration, see if we can converge on a preferred direction. Contrib maintainers who feel strongly about one framework over another will be free to deviate from that direction if they want to, but my guess is that if we pick a good library/framework, then most contrib maintainers will be ok with writing their client-side UI code with it. The sooner we pick that direction, the sooner that those contrib mainainers who don't have a strong desire to deviate from it can benefit from knowing what it is, and thereby use it when porting/creating their modules.

Note that one option in all of this is to keep Backbone and add Twig.js to begin migrating the inlined HTML-in-JS that we currently have. But I think that option should be explored alongside the other ones, like Ember+Handlebars, Ember+Twig, React+JSX, etc.

samuel.mortenson’s picture

So we know that Drupal 8 in its current form is compatible with some headless designs - why not have multiple teams work on example code for each framework we're interested in? Even if it doesn't lead to a decision, we would likely start to see patterns of limitation in Drupal core that could lead to important patches/issues for 8.x-1.x.

Each example project would need to meet certain requirements, some of which could be:

  1. Render a page with content regions based on Drupal's block placement. Block HTML can be rendered by Drupal for now.
  2. Render a Node with fields based on a template defined in Javascript. Field HTML can be rendered by Drupal, Node metadata should be rendered in the JS template.
  3. Render a form with basic submit functionality, all HTML should come from the frontend app.
  4. Demonstrate how a contributed module (another piece of JS) would extend or modify your code to customize templates, logic, etc.
  5. Demonstrate how a piece of content (region, block, node, form, etc.) would be re-loaded from the server after the page is loaded.

The code written wouldn't have to be perfect, or even work completely, but each requirement we come up with should have a detailed response and piece of example code for each framework. Of course the real requirements would be set by the community.

Edit: I didn't make it super clear originally - but the main difference between the above and what's in the issue summary is that 1) We should have set requirements for the teams building proof-of-concepts. 2) We can make real-world proof of concepts with the current release of Drupal 8, otherwise we may as well be looking at existing examples for the frameworks in consideration.

catch’s picture

@samuel.mortenson all those examples, apart from #3 and #4 in very limited cases, are for visitor-facing UIs. That's the sort of thing that should definitely be done in contrib, but it's not what the focus of this issue should be about (although the issue summary and original title do not help that by also including visitor-facing interfaces as examples).

samuel.mortenson’s picture

@catch Admin themes have regions where blocks are placed (#1), modules like Views render Nodes/fields in the Admin UI (#2), and reloading content via AJAX is an extremely common admin feature in Drupal 8 (#5). Also, if we're only thinking about the admin UI, where do admin-tools that live in the "visitor-facing ui" fit in? Thinking of things like Quickedit, Toolbar, and CKEditor.

netsensei’s picture

FWIW: here's my take on decoupled interfaces: www.colada.be/decoupled-drupal-a-couple-of-considerations.html

My blogpost highlights what Effugentsia states in #51 I considered a decoupled interface for a contrib module I maintained, two years ago, but didn't pull through because:

  • I had to choose a framework for my module while other contribs might have chosen other frameworks.
  • Without a common backend architecture, I had to create my own idiosyncratic API resulting in a lot of non-reusable boilerplate
  • If the community settled on a consensus solution 6 to 12 months down the line, I risked rewriting the entire thing to conform.

I do believe decoupled interfaces are the wave of the future. But I think we should be smart about where and why we implement them. I don't think just slapping a framework in Drupal and creating a generic backend API on functional quicksand is good idea. I see them more as reusable web components that enhance the UX instead of fully decoupled interfaces. I agree with Dries point of view that decoupled components should be limited to the administrative backend at first. In my view: to specific places at first even. That way, introducing decoupling becomes a manageable problem.

Also, adding a decoupled interface inevitably means revisiting the backend API. Almost all user input in Drupal is processed through the Form API going from a simple contact form to the entity display configuration forms. Can we refit/amend/repurpose the Form API so that it can work in a decoupled context too? If we want to build a foundation that can process user input in a uniform way, the Form API is an important part to consider in the equation.

chx’s picture

#55 is more about the other issue, especially #2645666-45: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS this comment from Wim.

While "the sad state of web development" blogpost are making the circles and was linked either here or the other issue, the answer to it has an incredible important and relevant insight:

Shiny new “products” rain from the Cloud almost everyday. [...] Its a good time to not be a fanboy.

I am not a JS person. Heaven forbid. If there's a worse language than PHP, JS it is. It was created to make the monkey dance when you moused over it and while it is slowly becoming sane(r) with ES6 (note the parallels of how PHP have crawled out of the pit and became a somewhat sane language over the years during 5.0-5.4) it is still a language I will only touch if forced. I'd rather go and line up at the soup kitchen than become a JS developer. Ahem.

With that said, Angular vs Angular v2 should make anyone pause. Say, Angular became popular about 2012 and you can toss your code with v2 coming this year if I am not mistaken. What you all will pick in this issue needs to be a "thing" for the next five or perhaps even more years. Yes, it's likely React will still be used in five years (or not. I posted about CanJS above which powered the Apple Store in late 2013 but if you try CAN.version today in the console, it doesn't seem to be any more and we are talking two years of a framework very much purported to be long term) but whether it's Perl-like has-been level of usage, who knows.

All I am saying: proceed with extreme caution. The change of pace in the JS world vs the change of pace in the Drupal world seems to be incompatible. I have no idea how we will resolve this.

Konstantin Komelin’s picture

I'd rather go and line up at the soup kitchen than become a JS developer.

Never say never, again :)

fago’s picture

That is the situation we have right now. For example, this line for adding an "Add" button in Views UI and this whole file of QuickEdit components. In other words, we already have UIs where we're not using a Drupal AJAX request and server-side rendering, because the latency of that would be silly.

So we are sending pre-rendered HTML to the client - I see no reason though this HTML could not be generated with the usual render/templating system. Sending it pre-rendered to the client is no excuse to bypass the theme system imo. But yeah, the tooling to easily do that in a proper way is not necessary there where it should be and we could improve our existing tooling to avoid a further split.

Agreed on optimistic feedback involving user feedback being impossible without duplicating/rewriting boatloads of things in JS.

Right, so to me this discussion boils down to re-implementing many things in node.js (#10725328) or have just stick with server-side rendering and improve the tooling / AJAX api around.

attiks’s picture

#49, #52I created 2 modules (Drupal 7 for now) to see how easy/hard it will be and to see what people expect from it.

- https://www.drupal.org/project/dream_permissions
- https://www.drupal.org/project/dream_comments

#56

All I am saying: proceed with extreme caution. The change of pace in the JS world vs the change of pace in the Drupal world seems to be incompatible. I have no idea how we will resolve this.

Fully agree, it changes way to fast for whatever reason. I think the only way to solve this is baby-steps, and adding a full blown framework cannot be the first step. Maybe a first step is to create separate issues and define a single use case in each so we have a list of pain points, this will make it easier to define what needs to be solved and how it might be solved.

#58 Yes for prerendered HTML by the theme, it is the only sane option, both for performance and accessibility reasons.

Yes to make it easier for javascript to interact with forms, even yes to a 'framework' that makes this easier. Drupal AJAX system has it merits, but it is not the most performant one.

The last year or so we started ripping out all javascript for the front-end and replace it where needed with custom code, the main reason is that a lot of contrib (and probably my own modules as well) is adding way to much javascript. We now use small libraries to add the functionality we need, so maybe it is an idea as well to investigate if this might help for other sites. Have javascript libraries for different use cases and make it easy for developers to use them, although this sounds a lot like components proposed elsewhere.

netsensei’s picture

Addition to #59, Here's a module I did for D7 with angular: https://github.com/netsensei/cats based on experiences with client projects. And this is a short screencast about above module: https://www.youtube.com/watch?v=p3zSQieBIe8 => garnered 7.800 views over 18 months. For a Drupal specific topic, I think that's a significant number.

For the Dutch speaking community: https://www.youtube.com/watch?v=sRva9hleFXY Presentation about a decoupled interface powered by a Drupal/SOLR backend. All Angular 1.x based.

Also, I fully agree with #56 and #59. Yes, I am excited about decoupled interfaces, but we should proceed with extreme care.

pbuyle’s picture

Issue summary: View changes

tl;dr: See https://twitter.com/pierrebuyle/status/687639939043975168

SAP and other complex UI are rightly called applications (here and mostly anywhere else). These are not just tin UI layer. Years of experience on applications development apply. We learned since long to create monolithic application tightly coupled to a specific framework and library. Even if Drupal was a rewritten as a Symfony or Laravel application, only few of us would expect things like Entity CRUD or Entity validation to be tied to the used framework. Instead, these would be properly encapsulated as decoupled components. Sure implementation classes may depends on Symfony's DI container and standard Symfony service. But it seems safe to assume that we would choose a framework that clearly favor decoupled components and architecture, in its design, documentation and practices. Yet this concern is completely absent in the initial reflexion.

There is a reason Drupal didn't switch to a PHP framework in all these year. And I don't think it was the amount of needed work for (not) rewriting it on framework X. I think we didn't do it because we didn't want to tie Drupal to any framework. So instead, Drupal was re-architectured around core concept and existing external components have been brought in, and some times enhanced.

Why should it be different for the front-end? Why should we pick a framework (some of the candidates are pretty opinionated and monolithic) and build (a better) Drupal (Admin UI) with it? Why is the discussion about picking a framework and not the needed font-end components that we need to build a better UI?

nod_’s picture

Let's assume we agree that the argument around this getting us more people is moot. If this is really about UX improvement we need to fix our UX process before, so that designers don't get burned out and our UX team plans not questioned in their entirety months after they've been agreed upon (that should be easier thanks to our regular releases).

Adding more very opinionated developers to UX issues is not going to help improve Drupal UX any faster. Any seemingly simple UX issue gets bikesheded to near-death, let alone complex ones. See issues for toolbar, module page, new content creation page, the search field on the module page (and this one is not even finished for some people) and let's not forget most issues from #2497361: [meta] Fix issues found during UMN Usability Testing 2015. From that list one, maybe two, would eventually be helped by a frontend framework but only if the framework integration has been done properly .

As it is now we already have jQuery, Backbone and Underscore loaded on every page for most power-users because of Toolbar. Adding another framework for admin UIs is just going to add a third big bundle of JS to admin pages. Killing what's left of usability of the Drupal admin of low powered phones (and that's what most users have). In the long term, getting standardized on a framework should mean we deprecated and remove jQuery, jQuery UI, Backbone, replace joyride, dialog, autocomplete, farbtastic and other utility libs, which in turn means rewriting quickedit and almost all other features. It's doable, sure, but there was the need of not wasting time expressed in this issue too. What Sally Young proposed in the lullabot podcast was the most sensible: If we really really want a framework, make a specific theme and tie the framework to it, not core modules.

After a lot of thoughts, reading back all of Dries' posts, listening to the latest lullabot podcast on this, reading posts from our folks I'm convinced that this is not the right solution to address the issues outlined which are: attract more devs, improve D8 UX, ride the wave of interest in JS (still need that issue summary update btw). Framworks are hot right now but don't address our problems and it shouldn't be our first step in the problem space of optimistic feedback and client-side templating.

As it was said before, most (all?) frontend people here think this is a bad idea — or at least a bad first step — I'd be nice if we didn't divitis our JS after all the work to streamline it.

nod_’s picture

Status: Active » Postponed (maintainer needs more info)
Issue tags:
Parent issue: » #2651660: Investigate where and how a frontend framework could be used

It's pretty clear some steps have been forgotten in this process hence i'm postponing this issue and opened #2651660: Investigate where and how a frontend framework could be used where we can talk about the why and how. Once this is clear — and the UX process part is figured out — we can start talking about technical solutions. Making any kind of decision before is premature, after all we can't know which critera are relevant before we know our requirements.

On a side note, taking from a relevant example on the performance aspect: Typo 3 Neos 2.1. The whole admin is EmberJS, single page app, inline editing, very fancy stuff. You get all that after the 2MB of JS are loaded (a lot of it is Aloha to be fair, but there is also jQuery, Backbone, jQuery UI & a bunch of plugins). On my cheap device the admin is not usable (it's a small phone, buttons overlapping all over the place), and it takes 20 seconds for the admin components to load (with warm caches), changing a page (within the single page app) takes about 9 seconds. On a more powerful tablet loading takes 13 seconds and page change 7 seconds. On my laptop the fan kicks in when I browse the admin. On the desktop as long as you're browsing admin screens that have nothing to do with the content, things are pretty fast. The catch is that those admin pages go to a new page, they're not part of the SPA. It's so different I'm not sure it's relevant to compare but if there is anyone to ask for feedback it's those folks.

( edit ) and they already bikesheded that for Typo3/Neos https://discuss.neos.io/t/rfc-js-refactoring-process-of-backend/283

jordan8037310’s picture

I agree with nod_, there is a gap in this selection process with where and how we are going about this, but I think there are some underlying discussions we should have instead of simply "which framework", and "what are the use cases". I want to break down a few points that I think will help the community approach this in a way that will benefit any front-end developer who is coming to work with Drupal as a decoupled framework.

Rather than focusing on a single solution, we should focus first on how we are going about deciding this, and with which core values or guiding principles we should use to judge proposed solutions.

In the context of this discussion, we should be focused on the problem of:

How can we dependably expose administrative forms in a way that allows for easy adoption of progressive decoupling?

Initial Thoughts

  1. People come to Drupal for a CMS experience, so our APIs are always going to be heavily coupled to the purpose of working with content—"content management".
  2. That means we need to be thinking of API contracts / conventions which are for the specific purpose of managing content, as opposed to driving an application.
  3. As a community, we need to adopt defining standards for decoupling. Instead of focusing on the which framework question, we should be discussing: how can we expose content management in Drupal to ALL frameworks. This presents the question of progressive decoupling in the context where we can focus on making consuming Drupal APIs pleasurable for all JS/Front-End developers instead of simply just one subset. Defining standards on how we can do this will likely be challenging, but worthwhile.
  4. Defining methodologies for decoupling in core should be advocated as a cultural practice. We can likely think up some kind of community slogan or initiative in order to encapsulate this ideology within Drupal. (e.g. "Proudly found elsewhere"), see list at the end for some ideas.

A Proposed "Drupal's Code of Decoupling" (to start)

  1. Always assume there will be a new JS framework in the future.
  2. Develop core content administrative APIs so that we can build Plugins on top of them for specific framework interfaces. Transformation into specific framework conventions can be adopted at the last phase before API rendering. This leaves the required room for framework/application specified interfaces.
  3. Choose a decoupling methodology that focuses on frameworks that leverage view components. [note: With web components somewhere on the horizon, we can expect the web to move progressively towards view-based rendering.]

I'm sure others have great ideas to add to the above, so I won't try to be exhaustive here.

How can we expose Content Management in Drupal to ALL Frameworks

  1. In order to expose Drupal as a strong content management tool for decoupled applications, we should first look at how we can create a common interface for administering content. Most of what we are talking about when we discuss content management is Forms and Views.
  2. Deciding on a convention to expose the Form API and Views UI seems to me like an easy 80/20 selection for where we should start.
  3. We should create a convention to expose components of the Form API either discreetly as a single object / payload.

Some bigger Decoupled Hurdles with Drupal's core Value Promises

What features/expectations do people have when choosing Drupal? How does delivering on these expectations become more complex when extending to a decoupled architecture?

I would argue that people choose Drupal for content management when they're looking for an extendable framework, but usually want these core features (or have expectations of these features being easily attainable):

  1. WYSIWYG control of content
  2. Fielded control of content
  3. path management (path,redirect)
  4. metadata
  5. layout control via UI (Panels)
  6. Robust content relationships (entityreference)
  7. Views building
  8. Cache invalidation management

Thoughts on Slogans for this initiative

  1. Proudly designed anywhere.
  2. Proudly rendered anywhere.
  3. Proudly experienced elsewhere.

I'm sure some of these lists could be broken out into their own issues, too.

nod_’s picture

This is very true. Your main question does start with a "how", it belongs in the other issue.

lakshminp’s picture

Agree with @jordan8037310.
First of all, not everyone wants a decoupled architecture out of Drupal. Among those who need one, its uncertain at this point whether they would be happy to use the framework dictated by Drupal. There are so many factors influencing this decision like business requirements, licencing, familiarity with the "chosen" framework etc.

We should rather focus our energies on building an interface which can expose the content under the hood in a framework agnostic way. This wouldn't change a lot for contrib modules too. It could be something like adding views support or features support for a contrib module. A contrib module maintainer could opt in to support this interface or otherwise. It could also come in as a feature request.

That said, I think a component based system would play nice with this. This is because Drupal's rendering mechanism closely resembles this kind of an architecture and the cognitive overhead would be lower if we follow this path.

pbuyle’s picture

I agree with jordan8037310's Drupal's Code of Decoupling. Starts by building APIs (server and browser side) to be consumed by frameworks powered applications. And, for a time, let contrib deal with building these applications (powered by these APis in core).

prestonso’s picture

Title: [META] Select a new client-side framework for Drupal core's admin UIs » [META] Supersede Backbone in core admin UIs with a new client-side framework
Issue summary: View changes
Issue tags: -Needs issue summary update

Thanks to all for the lively and fascinating discussion. IMHO, the insights that we are gathering on this issue and that we are all learning from are much more valuable and important than whatever result emerges from this debate.


As I clarified on the recent Lullabot podcast, this issue was purposely provocative to kick-start a broader array of discussions around the future of the Drupal front end, which is now demonstrated by #2645666: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS (which deals with BC-breaking changes in 9.x and beyond) and #2651660: Investigate where and how a frontend framework could be used. As @mrf (#31) and others suggested, I've updated the IS with a much narrower focus on why Backbone needs to be superseded (actual removal of this dependency from core would not take place within 8.x, as @xjm emphasized in #35) and which framework would be best-suited to do so. This narrower discussion is one we can still have while the "where/how" issue becomes clearer.

Now that this issue has evolved, several overarching themes surfaced which I want to summarize and address from my (own) perspective:

Progressive enhancement and graceful degradation

#6: No matter what Javascript-based fancying up we do, the page must still be usable without JS.

#12: I agree with the fundamental principle that an open web includes being open to people who wish to disable browser JS for whatever reason they have to do so.

Drupal has always been usable without JavaScript, and it's important that it remains that way. Disabling JavaScript should present the user with the UIs we already have in place rather than broken JavaScript-driven UIs. The "traditional" server-side approach to UIs should be kept as is, and, in fact, this is one of the key conceits of Dries' concept of progressive decoupling.

#19: Given one of my personal bugbears at the moment is staring at a blank screen on my phone waiting for js-heavy pages to download from certain websites when I’m out in the sticks, the very last thing I would want to be stuck with is needing to do a content update on a site whilst out and about and not being able to.

This is already a longstanding problem with our AJAX framework. Particularly for use cases like the one above, I would advocate some kind of logic allowing users in more spotty network conditions to receive the minimal HTML version of particularly rich UIs (#26: "cutting the mustard"). Alternatively, perhaps it's better to make Drupal more offline-first through techniques such as ServiceWorkers.

Twig and shared templates

#11: How is this going to work? Some sort of JS templates for the admin side, and a completely different set of Twig templates for user-facing pages? People now need to two completely different way to work with Drupal markup?

#21: So we have to learn twig now just to drop it on next Drupal?

"Shared templates between client and server" seems to be a hard requirement. In my opinion, given that Twig is the official templating language for 8.x and especially that so much invaluable work went into making Twig in Drupal a reality, any client-side framework that is adopted should also leverage Twig. Many frameworks can use different templating languages than their own, including through a parser for Twig or through a transpiler that converts Twig to the framework's language of choice (less desirable).

#45: So wouldn't we end up with two different, rather complex systems that have to be properly integrated to work properly together?

I agree that this is absolutely a key concern, but it seems that if we really desire complete unity between rendering across the server and client, we would want a language that operates across both. Currently, the only such option that provides that universality is JavaScript (or, arguably, Dart). So, the question we need to answer is whether unity across templates for themers ultimately outweighs unity across rendering systems for all developers.

Client-side framework longevity

#8: Just as Backbone is now passe in the eyes of many, I expect every library listed above, including whichever one "wins", to be passe within 24 months at most.

#30: Sounds too risky to bet on a big strongly opinionated JS framework at this point.

#56: The change of pace in the JS world vs the change of pace in the Drupal world seems to be incompatible.

As @chx (#32) and others mentioned, the landscape for client-side frameworks is in constant upheaval (the volatility of their toolchains notwithstanding), and any framework also leads to at least some level of lock-in (#20). @techlust makes an excellent point in #33 that the extensive corporate backing and large communities many frameworks currently enjoy — and the number of companies racing to adopt them — should not be discounted. We should at the very least kick off this discussion while the JavaScript community continues consolidating.

Other existing solutions

#20: I don't see the problem, nor anything there that cannot be done with current AJAX framework, maybe some improvements to that would be fine but that's it.

#47: Using the AJAX framework doesn't get rid of latency: you always need a round trip. That's the biggest problem. ... To eliminate network latency, you must run the logic on the client side, which requires JavaScript.

#59: Drupal AJAX system has it merits, but it is not the most performant one.

Optimistic feedback (especially instantaneous render and preview) typically involves some client-side templating and rendering. Non-blocking user workflows are not easy to implement through our existing AJAX framework, which has served us eminently well but suffers from race conditions when many operations are performed in quick succession (Views UI is particularly vulnerable) and incurs server latency. A server-side solution such as WebSockets would require rethinking server architecture and likely just as intense a paradigm shift as this one.

Selecting a framework in contrib instead of core

#18: If core provides a solid "traditional" approach with the right hooks, then contrib-based admin themes can experiment, evolve, and Darwin it out while still providing that more integrated, Facebook-esque experience.

#22: The client-side framework selection will never happen in contrib, for fear of conflict, and for fear of losing convenience ("To install Media you must first install the Ember module").

#51: The sooner we pick [a client-side framework], the sooner that those contrib mainainers who don't have a strong desire to deviate from it can benefit from knowing what it is, and thereby use it when porting/creating their modules.

Deferring this decision to contrib for the ecosystem to duke it out means that Drupal is beholden to the larger JavaScript landscape. As soon as one consensus begins to form around a framework, another project could emerge that upends that consensus, leading to wasted time and technical debt. By adopting a framework early into Drupal core, we can codify best practices using a sample implementation which could then guide other implementations in contrib. Multiple frameworks should not be loaded onto a single page simply because two contrib modules decided differently.

Authenticated versus anonymous UIs

#11: What about UI components that appear for authenticated users on user-facing pages, like Toolbar, Quick Edit and Contextual Links. Would those use this JS framework?

#36: So in general, we aim to serve anonymous page requests without any JavaScript at all by default, and I think we should keep that aim.

I agree that no unnecessary JavaScript should be loaded on anonymous user pages unless desired for specific forms where that level of interactivity is truly needed. However, for authenticated users, a framework could be reused across Toolbar, Quickedit, Contextual Links, and more. I am on board with @catch in limiting this specific issue to site building and content administration interfaces, with the caveat that the aforementioned authenticated user experiences outside the confines of the admin theme would also benefit.

Decoupled "front-end components"

#17: IMHO, Drupal core needs to standardize on a good font-end DI container, (p olyfills for) standard language features (eg. ES6 module loading) and APIs (eg. Fetch API for HTTP request) and lightweight decoupled front-end components.

#55: I see [decoupled interfaces] more as reusable web components that enhance the UX instead of fully decoupled interfaces.

I think we need some concrete examples of how this would work or has been done by others in a separate issue (related: #1804488: [meta] Introduce a Theme Component Library, #2289619: Add a new framework base theme to Drupal core).

prestonso’s picture

Status: Postponed (maintainer needs more info) » Active

Restoring status now that the IS is updated.

nod_’s picture

Thanks for the update. I would argue that "It would also accelerate UX improvements by providing better code reusability across all of our UIs" still isn't true, frontend code is not the bottleneck on UX improvements, never was.

All in all the situation hasn't changed much so I'd say this should still be postponed. We still don't have compelling reasons to consider adding a framework (some things are talked about on twitter but still not in the issues so they don't count). If there is a plan for features or something that really really needs a framework just tell us. I'm all for getting better tools but not for no reasons.

Quickedit is doing it's job fine with backbone, toolbar shouldn't need it to begin with (let alone another framework), contextual links, meh, either way is fine. If what you want if a framework name without even thinking about what we'll do with it and how, then it really doesn't matter which one we choose. My vote would go to the most consistent and opinionated one, ember since I don't look forward to coming up with guidelines and recommendation for a framework we're not actually using. We can do solution #2 of comment #38 and be done with it.

pbuyle’s picture

On the, Decoupled "front-end components". Issues #1804488: [meta] Introduce a Theme Component Library amd #2289619: Add a new framework base theme to Drupal core are interesting, except the front-end the components I was talking about are not about theming at all and should not be viewed as a theme thing. If we want to move forward with Drupal UX (not just UI, and not just its look), we need to differentiated between theming and front-end code. I may be wrong, but IMHO, the most widely used meaning of "theme" for computer software is the "graphical appearance" of the software. The client side components I'm talking about have nothing to do with the graphical appearance.

The kind of components I was talking about are things like a Entity CRUD component (ie. an object responsible for client-side Entity CRUD operations, hidding the gruesome details of communication withs the server) or an Entity Validation component (ie. an object responsible for client-side validation of an entity object). I guess I should have used "services" instead of "components" when I wrote "lightweight decoupled front-end components". But I didn't use "services" on purpose, because these are not interface to web services, neither do them have to communicate with the server in order to do their job. And they don't have to have any behavior at all, they could be immutable collections of data (for instance, configuration variables used by other components like the list. Because yes, you can use DI to inject your configuration variables).

If this existence of this kind of components in Drupal core needs to be discussed in a separated issue, I doubt #1804488 or #2289619 are the ones to use.

effulgentsia’s picture

frontend code is not the bottleneck on UX improvements, never was

Are you sure about this? While I agree that it's not the only bottleneck, my sense is that it is a significant barrier. Because to improve UX, ideas need to be tried out. But trying an idea out requires some implementation of it. And if creating that implementation is not enjoyable, who's gonna do it? Sure, maybe eventually someone turns up, but probably not as quickly as it would happen if it's a fun project. Especially since the implementation may well end up being thrown out (e.g., upon some real world testing of it, it turns out to not work very well from a UX standpoint), how satisfying it is to code it is an important factor.

Take for example, tabledrag.js. That's 1500 lines (to be fair, including docs) of some pretty scary looking code to me. Huge props to everyone who worked on this component over the last decade, with no library assistance other than from jQuery. But I wonder if, for example, something like React would make all the manual binding and triggering unnecessary, and reduce the code to something much friendlier. Maybe it wouldn't, but maybe it's an example of something worth exploring to find out.

And if just something like a tabledrag component requires this much of our own in-house code, what's the barrier to exploring even more sophisticated UIs?

As a (very imperfect) analogy, look at what happened when we moved markup from PHP files (tpl.php templates and theme functions) to Twig. By making working with markup more enjoyable, we got an order of magnitude more front-end developers improving Drupal core's markup during 8.x development than we had during 7.x development. And so, our markup became much better. We didn't need Twig to have better markup, but in practice, it helped by making the work of improving the markup more enjoyable.

Now, I'm not saying we should just blindly pick whichever framework is most popular solely to attract some JS developers to Drupal contribution, just like we didn't pick Twig for being the most popular template language (since it's not). Instead, we should look at the real merits and drawbacks of what's currently out there (frameworks, libraries, etc.) and evaluate whether there's something which has more benefits than drawbacks with respect to our needs. The answer may end up being that there isn't yet, but how will we know without exploring, and this issue can serve to capture people's knowledge that's relevant to answering that question. Or maybe #2651660: Investigate where and how a frontend framework could be used is the better issue for that, I don't know.

chx’s picture

If you want to go down the Twig route then pick personas -- if they have real life equivalents even better -- and figure out what they need and then pick your framework that way. Expect a fight. Do NOT let yourself derailed.

nod_’s picture

Picking personas sounds like a great idea. This issue is frustrating because there are valid arguments for this but they need to get pulled out one by one. And indeed, that would be better to discuss in the other issue.

As for a quick reply on tabledrag that's the one component that might get simplified, it's true it's a pain. Then again we don't have a new tabledrag UX to investigate, no concrete plans to change it from UX team. Since Munich I wasted 3-4 days worth of sprints during events and more at home trying to make a replacement for it. The code was a pain but no UX direction meant I wasted time on useless aspects as well and it ended up not going anywhere. Even if we have a library/framework that is nice to work with, people will make their demo and wait for month for anything to happen because UX issues are still a mess because our process for them doesn't work. Then they won't be making a second demo.

For demos people can pick whatever they want to write their prototype, it's not like we prevent them from using any library in their experiments so I feel this is not that strong an argument. Maybe people don't dare making a demo with something else, but a lot of people don't dare touch core to begin with.

I really want to get behind this issue but it's hard when justification is done piece by piece. With all that Dries wrote about it I'd expect something more fleshed out.

mdrummond’s picture

I wasn't very familiar with Ember, the framework Dries indicated seemed the best fit after the initial analysis, so this week I completed a course on Code School to learn more (https://www.codeschool.com/courses/warming-up-with-ember-js). After I finished, I read through the Ember version update notes for the last couple years on the Ember blog (http://emberjs.com/blog/).

Many of the core techniques I learned about from the Ember course have changed in the couple years since that course was created. The Ember project appears to try to make changes gradually, with sensible deprecation warnings and semantic versioning. Their community appears to care about stability, but even so keeping an Ember app up to date would require a decent amount of work to handle all the deprecations between the 1.x and 2.x versions.

Anyhow, what struck me is that we're talking about these client-side frameworks like something we pop in and then we can make a few forms here and there snappier.

That is not the impression I got from the little that I've learned of Ember. It seemed like a nice enough framework, but it seems very much geared towards handling the routing and templating of an entire site, not bits here and there.

There's been a lot of talk about how it's important for our interfaces to render on the server first, then get quick updates based on user interactions. Ember does not seem compatible with initial rendering via Twig, and then just swapping in Ember templates when updates are needed. If we want server rendering, that means using Fastboot, Ember's isomorphic JS efforts. Fastboot isn't entirely stable yet, it seems, but it looks like that is right around the corner. Anyhow, if you want Fastboot, that means using Node.JS.

Ember also seems pretty tightly coupled with HTMLBars, which seems like an updated version of Handlebars from what I can tell. The syntax is pretty similar to Twig, to the point where I think it would be very difficult to use HTMLBars for one part of the page and Twig for another. Both use the {{ variable }} syntax, for example.

It also really seemed like Ember was geared around using its routing system from a single index page, with virtual paths like admin#/structure/content-types used to route in controllers/components.

Another thing to note is that Ember can now do the sort of virtual DOM diffing that React is known for through Ember's new Glimmer engine. This is what allows for the snappy updates that seems to be the goal. That means having JSON with the data needed for these updates, along with the virtual representation of the DOM, stored in client-side memory. So that means fast updates... if the device has the power to handle loading everything. We'd need testing, of course, but I'd have to think that on mobile browsers, in particular, that could be a challenge. We had an entire mobile initiative so D8 admin could work anywhere... I would be concerned how possible that would be with a heavy framework.

It is entirely possible that I am not understanding the full breadth of how Ember can be used. But to me, this does not seem like a trivial amount of work. I have to think the other monolithic JS frameworks that are being considered are going to have similar issues. It may be much more difficult than expected to use a monolithic framework to just replace bits and bobs of the admin UI here and there. I'm skeptical that we could use something like Ember for front-end admin components like Toolbar, Quick Edit and Contextual Links (if we are replacing Backbone) without a lot of challenges.

We're already using a fair amount of JS for our admin. Replacing that with something else doesn't seem unreasonable. But I do think this could end up being a ton of work for a somewhat marginal benefit. I do understand that some of the modern PHP techniques that could allow for faster interactions like ReactPHP and Icicle have server infrastructure implications. But from what I've seen so far, the same is true for something like Ember. If we really do care about content being rendered on the server, Node.JS will very likely be needed for that.

I'm trying to keep an open mind, and I am sure all have the best of intentions towards improving Drupal's admin interfaces. I can't help but feel like we are trying to find a justification for a solution before we truly examine the goals and user needs that would require that solution. I don't think a monolithic client-side framework is a cure all. I also really wish people would stop citing a JS framework as a recruiting tool for front-enders to help with Drupal.

Maybe including one particular JS framework gets people in that framework community excited, but people who prefer another JS framework? Not as much. I'm also not convinced that including a JS framework in a PHP project will necessarily get JS developers involved, particularly when our community isn't on GitHub, like most of the JS community is used to. And not all front-end developers love working with JS frameworks, for some they can be intimidating.

For me, this whole issue has sucked away a lot of the energy and excitement I have had towards Drupal 8. The uncertainty this is generating is draining my personal desire to work on Drupal 8 issues, which is unfortunate since we are in the window where 8.1 improvements can be made.

Emotion is hard to get across in posts like this, so I want to be explicit that I'm not upset or anything, I'm just feeling drained and disheartened about all of this. Anyhow, that's my two cents for what it's worth.

mdrummond’s picture

Just came across this article on RAIL, a way to talk about performance metrics: https://www.smashingmagazine.com/2015/10/rail-user-centric-model-perform...
- R: Response (respond to actions under 100ms)
- A: Animation (complete animation frames in 16ms)
- I: Idle (prep optimistic ui in chunks of 50 ms)
- L: Load (load page under 1 second)

This also links to a nice explanation of optimistic UI: http://info.meteor.com/blog/optimistic-ui-with-meteor-latency-compensation

These are exactly the sort of goals that I think are why this discussion is taking place.

For example: "Drupal would be easier to use if performance seemed seamless" with RAIL as metrics used to assess whether or not we are achieving that goal.

What bothers me is that we have jumped right to the notion that we need a client-side framework, which may or may not help achieve that goal. Heavy frameworks may increase mobile load time, for example. First figure out what we are trying to achieve, then look at the different ways we might achieve those goals, then once we settle on an approach, figure out the exact tactics we will use with that approach.

Most importantly, get community buy in each step of the way. Making large changes to a huge chunk of the way Drupal works will take a lot of people to make that happen. If the community is not on board, who will do that work? This is a volunteer effort. Yes, the project lead can make decisions for Drupal, but if the community is not on board with that vision, work will be difficult.

Finally, weigh the level of effort required to commit to this undertaking against the possible benefits, as well as what that means not doing instead. There is only so much we can do at any one time. Getting the admin side of Drupal to meet these sort of RAIL goals is maybe good, but how much effort will that take? What will we not have time to refine in the Drupal 8 cycle instead?

I'm sure there are some that will feel that the resistance to this push for a framework is mere bikeshedding, but I feel like we're being asked what color to paint the bike shed before we've decided whether it's even a good idea to have a bike shed in the first place.

I feel confident that the Drupal community when we work together can do really great things. I would really like to see us to do the hard work as a community to really examine the underlying issues that are driving this initiative before pushing forward. I'm certainly feeling some burnout from this, and I suspect others are as well. If we take a step back to think things through, I think we can stay more energized and positive.

Anyhow, Drupal hugs to you all.

effulgentsia’s picture

this does not seem like a trivial amount of work

Absolutely right. Refactoring Drupal on top of Symfony components was not trivial. Refactoring theming around Twig was not trivial. Heck, even adding CKEditor to core was far from trivial.

this could end up being a ton of work for a somewhat marginal benefit

People raised the same concern for Symfony, Twig, Wysiwyg, and many other large changes. And they were right to do so, just as you're right for raising that question for this issue. There still needs to be more clarity and precision around what the benefits will be. The discussion in this issue and the related issues is about figuring that out. It took similar discussion to hash that out for Symfony/Twig/CKEditor/etc.

For me, this whole issue has sucked away a lot of the energy and excitement I have had towards Drupal 8. The uncertainty this is generating is draining my personal desire to work on Drupal 8 issues, which is unfortunate since we are in the window where 8.1 improvements can be made.

I'm really sorry to read that. I hope that ends up being a short blip for you, and that you'll rediscover your enjoyment of Drupal 8 soon. But I don't think we should shy away from discussions of important happenings in our industry and if/how/when Drupal should incorporate them. Projects that do shy away from that don't survive and thrive for 15 years, like Drupal has. I agree with you that there's a lot of uncertainty about this entire problem-space, but I think that's all the more reason to be discussing it widely, via a public issue. I really do hope that you'll find the right balance for yourself between working on more well-defined Drupal 8 improvements that you care about and offering your insights to this more chaotic discussion.

What bothers me is that we have jumped right to the notion that we need a client-side framework, which may or may not help achieve that goal...I feel like we're being asked what color to paint the bike shed before we've decided whether it's even a good idea to have a bike shed in the first place.

I think that's a fair criticism of this issue's original title, but I think the new issue title as of #68 and the corresponding comment is a start at incorporating that feedback. As with any issue, the resolution of this one may well end up being "Won't fix" if there turns out not to be a sufficiently compelling case to be made for it.

Drupal hugs to you all

And to you. Thanks for taking the time to write up your insights and concerns.

chx’s picture

this could end up being a ton of work for a somewhat marginal benefit. People raised the same concern for Symfony

-- and in the case of Symfony CMF they were totally right. Since the removal of the same path serving different pages for different mime types (html, json etc) the new system offers at most marginal benefits while it's infinitely more complex and scales much worse (but even before that removal the costs far outweighted any benefits). Making routes the heart of everything led to a truly bad and byzantine DX which is hardly surprising because neither incoming nor outgoing links are routes so there's a lot of unnecessary conversion going on. My heart broke the other day when someone said on IRC "cache rebuild on a d8 site with 3000 menu links takes 3 minutes " -- the D6/D7 menu link system has been tested for hundreds of thousands links.

So: tread carefully. Change for the sake of change doesn't always work out.

catch’s picture

@mdrummond thanks for the post and looking into ember properly, that helps to frame things a bit.

@effulgentsia:

People raised the same concern for Symfony, Twig, Wysiwyg, and many other large changes.

Well as chx says, Symfony has been a mixed bag. Refactoring on top of some components worked out fine, others created massive amounts of technical debt that has (is) taking months/years to resolve, some were in-between. In at least some cases, the most troublesome conversions happened due to insufficient requirements gathering. The issue was not just that there was a lot of work involved, it was that some of the work done was unnecessarily, or required multiple amounts of additional work later in the cycle to get it releasable.

For example see my comment from October 2012 #1793520-7: Add access control mechanism for new router system pointing out that access requirements for the new routing system had not been evaluated properly (i.e the quite basic point that menu link access depends on routing access in Drupal 7 and earlier and this was not a requirement that could be ditched). This was all before the main routing patch went in, and the fallout from those patches led us all the way up to #2339219: [meta] Finalize URL generation API (naming, docs, deprecation) and many release blocking issues in-between, because we went ahead anyway without stepping back and evaluating whether what was being proposed was actually fit for purpose and where it would eventually lead.

Then similar themes are repeated in #1447712: Evaluate Symfony form component as a replacement for Drupal FAPI which is being discussed this week, where there hasn't been a single requirements-based justification for using Symfony Forms at all.

WYSIWYG is a bit different. I don't remember people objecting to the amount of work involved in adding WYSIWYG to core (although similarly, not many volunteers to do that work). In previous years there's been a lot of push-back against using an editor at all, but that was a conceptual discussion (this issue and the node one reminded me of http://drupal.org/project/live, which was on groups.drupal.org for a time, and was supposed to be a WYSIWYG killer by actually rendering as you type).

The concerns during the Drupal 8 cycle were more about properly integrating any wysiwyg editor with the text format system, and this being a pre-requisite to adding the editor. Since this refactoring actually happened, in the end there was not a lot of controversy about adding the editor itself - it's optional, and core's architecture supports it as well as we could have anticipated. i.e. it was done properly, in a way that did not cause widespread objection. Concerns were less about the amount of work, but that it actually got done, as opposed to slapping a wysiwyg on top of an inadequate system (with the UX and security fallout that would have resulted from that).

Then with Twig, I think everyone knew it would be a massive amount of work, but the sheer amount of work and the relatively late start, meant at the end we were converting individual core admin tables to individual Twig templates just to get them done, which was the exact opposite direction of issues like #1804488: [meta] Introduce a Theme Component Library and #1876712: [meta] Convert all tables in core to new #type 'table'. Many things which should have been consolidated/refactored/reorganised, had to be done as one-to-one conversions to get 8.x releasable. Other ideas like mostly deprecating preprocess ended up getting more or less abandoned, and we still pass $teaser and $page variables to node templates in addition to view mode, actual theme-level infrastructure didn't really get modernised etc. - again because we had serious nuts and bolts release blocking issues to resolve like autoescape.

Back to @mdrummond:

I would really like to see us to do the hard work as a community to really examine the underlying issues that are driving this initiative before pushing forward. I'm certainly feeling some burnout from this, and I suspect others are as well. If we take a step back to think things through, I think we can stay more energized and positive.

To me this sounds like:

1. A higher-level discussion of what exactly Drupal 8's UX priorities should be (going back through usability test results. IA and terminology? Content administration? Field UI? Blocks? All of them?).

2. A bit of retrospective on the 8.x cycle up to now, and what exactly the current state of subsystems which would be affected by this is (existing js, AJAX framework, Twig, routing, rendering, Form API). Also examining relatively quick wins like removing the full server-side render + http redirect we do for all non-AJAX form submissions, which already has an issue open.

3. A commitment to tackling the underlying issues (as was done with WYSIWYG) as a pre-requisite to implementing new features. Whether those are technical (js testing?) or UX discovery, before jumping ahead with implementation of new and shiny things. Lewis Nyman's blog post covered this well.

mdrummond's feeling of burnout reminds me of the massive burnout I experienced when Drupal 8 was branched at DrupalCon Chicago. Fortunately Drupal 8's critical issues are in a much healthier state than 7.x was at this same point after release, and we've committed to not opening Drupal 9 for some time. But there's still a lot of things left over from beta/rc that we're only just in a position to circle back on now. The feeling of being pushed ahead while both contributors and the code base itself are still trying to recover (or 'being asked what colour to paint the bikeshed before we know if we need a bikeshed') is not a good one.

Wim Leers’s picture

I found myself nodding along with every paragraph @catch wrote. Thanks for writing that, catch. It's a great description because it has the needed nuance yet remains sufficiently high-level.

Drupal hugs to you all.


I have only two constructive things to add, both only tangentially related:

  1. @mdrummond: First: extra Drupal hugs for you! Second: I totally understand you feel drained. If/when you want to, let's continue our collaboration on #2061377: Allow image style to be selected in Text Editor's image dialog (necessary for structured content) to get that moving forward. That'll result in tangible improvements in 8.1 :) I know it'll feel great to get that done for me personally, and I think it will also feel great for you.
  2. @catch: RE: Also examining relatively quick wins like […]: I'd love to see what PJAX/Turbolinks would buy us in terms of making the UX faster: #2603794: Add PJAX/Turbolinks support. I think I'll be able to work on that somewhere in the next few weeks. It could of course end up being a marginal improvement. Let's find out.
webchick’s picture

1. A higher-level discussion of what exactly Drupal 8's UX priorities should be (going back through usability test results. IA and terminology? Content administration? Field UI? Blocks? All of them?).

We have a pretty complete (and prioritized) list over at #2497361: [meta] Fix issues found during UMN Usability Testing 2015, but honestly, if you've seen any of the usability testing results in the past 10 years or so, it's exactly the same boring, tired issues:

  • Our terminology is completely arcane and causes Drupal to appear frightfully intimidating to normal web developers. (Not helped one way or the other by JS, will have to be targeted for D9; at best we might be able to play around with a contrib translation in D8.)
  • Wayfinding is difficult to impossible. If you want to affect something on the page here, you have to somehow find your way to a completely other page in the admin interface there. — This could be helped a lot by JS, e.g. by expanding places where in-place editing is incorporated.
  • Normal people have a huge reliance on previews, esp. when learning a new system, for reassurance they're doing the right thing, and Drupal shuns previews at every turn, except in very specific circumstances (node/comment forms, block UI (if you manage to find the special link, Colour module, Views (if you manage to scroll down enough to find it) - JS could help tremendously here, esp. if we start rebuilding UIs around "preview first," which would definitely be the preferred/modern approach.
  • An overwhelming number of choices is presented to the user at every turn, with no clear hierarchy as to the importance of the options, thus the user is either paralyzed by choice or spends ages independently evaluating everything to see if it's important or not — Does touch JS, e.g. progressive disclosure
  • The "russian dolls" / "order of operations is backwards" problem — in order to build a "page," you need to build some discrete bits of content, which is placed into some number of views, which is placed into some number of blocks, which is placed into some number of regions, which is placed into some number of themes. JS could help a lot with this as well, by shooting off to the relevant APIs but keeping a single central UI/dashboard for building things out (or not, but at least we could explore what UIs like this could look like).
  • There's no example content in Standard profile which leaves new users baffled as to where to start — Not a JS problem.
  • Most problematic UIs in D8: block UI (regressed from D7), field UI (regressed from D7), Views (always confusing :D). All of which could benefit from JS, esp. Block/Field which currently take like 45 clicks to get a simple thing done.

And frankly, I am sick of hearing about these issues. Usability testing Drupal has actually become super boring. We already know all the shit we need to fix, because it's the same shit we've needed to fix for the last decade. I'm sick and tired of looking at something like SquareSpace and feeling envious and sad. There's no reason Drupal can't have a UI as beautiful and elegant as the API under the hood. We need to remove blockers to getting it done.

Part of those are indeed process-related blockers, and that's definitely on my radar and I've been working with Bojhan and others on some proposals (which sadly got completely derailed due to the untimely end of my 17 year marriage over the past few months :\) to help with that. See the end of https://events.drupal.org/barcelona2015/sessions/making-drupal-better-ou... for a sneak preview.

But I think another huge part is that if you talk to your average JS developer (and Drupal core developers who are fluent in JS are not your average JS developers ;)) they wouldn't touch something like Drupal with a 50-foot barge pole. Our main JS APIs (e.g. states, AJAX, etc.) actually go significantly out of our way to not write any JS code if you can possibly help it, in favour of Drupalicious #properties. :P Because we're largely PHP developers, and why would we want to touch that smelly stuff? ;)

But that's not the way the web is trending. Normal web developers would look at the set of UX problems we have and the set of idealistic UIs we'd like to build, and reach for better tools. I think it's perfectly fine to explore what those better tools might/could look like while we're also attacking process problems and also coming up with next-gen UIs. We definitely don't want to waterfall this, or D9 UX testing is going to be just as boring as D6/D7/D8. :P

mrf’s picture

This comment sent me down memory lane.

if you talk to your average JS developer (and Drupal core developers who are fluent in JS are not your average JS developers ;)) they wouldn't touch something like Drupal with a 50-foot barge pole. Our main JS APIs (e.g. states, AJAX, etc.) actually go significantly out of our way to not write any JS code if you can possibly help it, in favour of Drupalicious #properties. :P Because we're largely PHP developers, and why would we want to touch that smelly stuff? ;)

Off the top of my head I can think of two full-time Drupal developers I have worked with who have left the Drupal world completely in order to write Javascript all day every day. Nevermind the mass exodus of Development Seed.

To me convincing this group to come back to Drupal feels like an impossible task, and would be a far easier task than convincing someone entrenched in the Javascript world who has never heard of Drupal to give Drupal a chance. No framework choice solves this problem, because it is not entirely a technical best-tool-for-the-job problem.

We need to face the hard reality that there are only two ways to get entrenched Javascript devs to use Drupal.
#1 Convince their boss that Drupal is a necessary part of the stack that needs to be forced on them.
#2 Focus on making Drupal a super-robust (and speedy) data-serving machine so that these devs don't even notice that it is there (this helps with #1).

#1 won't make us a lot of new friends but #2 definitely could.

Moving a big chunk of our JS code to a framework will not bring members of that framework community into ours to come solve our problems, especially if that code is still all sitting over here in our sandbox, but it MIGHT make some of our existing front end developers not rage-quit Drupal.

Keeping this small and dwindling audience happy is my personal number one priority in this decision. If we start to see a lot of excitement from the front-end Drupal community (like we did with Twig) I'm all for taking on any amount of work to make this happen. Right now I'm not seeing or hearing any level of excitement about this move.

webchick’s picture

See, I come at it from a different angle. If we acknowledge that the Development Seeds of the world, and the friends both you and I know who have left Drupal to pursue JS are not just weird "out there" flukey people but instead representative of the way the entire web is actually trending, then we have no choice but to make moves that make Drupal friendlier to non-Drupal JS developers. (Not in the hopes of attracting magic new contributors, which keeps getting bandied about for some reason, but just as a matter of survival for our project.) Just as we had no choice but to make Drupal easier and easier to use for content authors/site builders, as our competitors do the same. Otherwise, our project becomes a relic and one of the core strengths of Drupal has traditionally been not being afraid to stare down and tackle tough problems and keeping on the cutting edge of technology. I don't want to see us lose that.

And to be 100% clear, no one here is out to burn anyone else in the context of this discussion, so comments from Marc and other contributors who've given their all to D8 and feel this way do make me sad and worry. :( But at the same time, we cannot possibly make a fundamental technical decision for our project based on emotional arguments. We do need to acknowledge the concerns being brought up in this discussion (and elsewhere) as valid (and there are a lot of such valid concerns), and I feel like we are all trying to do that: to really listen to what one another has to say. And, to Dries's credit, the whole progressive decoupling discussion is an attempt to appease both audiences and gain the best of both worlds—client-side friendliness (including for fully-decoupled architectures) + not throwing our baby out with the bathwater. So I'm not understanding the resistance to at least exploring this.

mrf’s picture

I don't think there is resistance to exploration, but I am hearing a lot of fear about yet another front-end decision being foisted onto front-end devs by people that don't spend much if any of their time working in that layer.

I'm all for people passionate about the front-end of Drupal and well versed about these frameworks making an educated choice about how Drupal competes and stays relevant in this landscape. I just want to make sure that front-end devs are well represented in this discussion and driving the decision. As someone who has been out of the front-end game for a number of years I don't pretend to have the answers here, but can empathize with how they are feeling having been on the outside of these types of issues looking in in the past.

Letting front-end devs guide the decision is the only way we avoid making more Drupalicious decisions that drive front-end devs to start framework shopping on their own.

If you identify as a front-end dev, please speak up! I can only hope you are aware of this issue and are following along with the rest of us.

nod_’s picture

Reading the comments… I declare the issue summary off topic :þ

Read comment below instead of this one, much better than what I wrote.

I think we're all fine with exploring things, staying relevant technically and not bumming people out.

So I'm not understanding the resistance to at least exploring this.

The issue did not start as an exploration: "Select a new client-side framework for Drupal core" is some Rambo-grade explorating. Follow that by barely satisfactory arguments being given one by one and a little all over the place, it does not inspire confidence and make it look like a fearful reaction to… something — especially since apparently we "need" to get something in "fast" — so we definitely agree with you and: But at the same time, we cannot possibly make a fundamental technical decision for our project based on emotional arguments. Progressive decoupling is a nice idea but why select a framework before we even have a proof of concept somewhere so know what we should be careful of? There is something we absolutely have to waterfall and it's figuring out Why/How/Where before What, since it sucks to work in a vacuum.

I don't think anyone is dead set against progressive decoupling, but please at least let's agree on which kind of hammer we need before we discuss which brand we want to use.

akalata’s picture

Adding thoughts from my perspective as an infantry-level contributor, who works towards executing the strategic and tactical decisions determined by core maintainers, initiative leads, and sub-system maintainers:

Answering @webchick in #83:

So I'm not understanding the resistance to at least exploring this.

@mrf's response is #84 is a good start:

I don't think there is resistance to exploration, but I am hearing a lot of fear about yet another front-end decision being foisted onto front-end devs by people that don't spend much if any of their time working in that layer.

For me, the immediate reaction is resistance because the initial proposals are not phrased as questions ("What are some ways to solve X with Y?" or "Could X be solved with Y?"), but as foregone conclusions - "Add X which solves Y". Which leads to the response "Is X the best solution for Y?", which then appears to be resistance but is really the question we need to be discussing.

"Add X to do Y" is a good title for a task when the solution has buy-in and has been vetted by the whole range of stakeholders (from BDFL to the volunteers expected to be doing the work), but it's not so great when it appears (rightly or not) to come out of the blue. A better title might have been "Do Y" - which would initiate discussion of how to solve a particular problem and not immediately polarizing people into being for or against a particular action.

markabur’s picture

representative of the way the entire web is actually trending

There's more to the web than Github. Since 2008 I've been building primarily marketing/communications and academic websites using Drupal exclusively and don't have a single project hosted on Github. Where is HTML and CSS on that graph?

#2 Focus on making Drupal a super-robust (and speedy) data-serving machine so that these devs don't even notice that it is there (this helps with #1).

Isn't that already one of the benefits of D8 today? Sorry if I'm mistaken -- I haven't been following D8 development that closely.

To me, Drupal is a fantastic content management system, especially for websites that need richly structured content. I'd wager that the main reasons entrenched JavaScript developers don't want to use Drupal is because they like working on projects that don't need a CMS, and they don't care about richly structured content. Didn't DevSeed leave Drupal because they enjoyed working on non-CMS projects, and using a static website generator for content when needed? That's fine -- it's just a different sort of project than the typical CMS-based marketing site that Drupal is so good for. If I ever need to build a non-CMS web project I probably won't use Drupal. Isn't that OK?

mdrummond’s picture

I appreciate the kind comments and personal concern shown in this thread. However, I also shared technical concerns and thoughts that I think are worth restating:

  • If we want client-side responsiveness (less than 100ms for example) in addition to fast page loads (particularly on mobile), that likely means we need isomorphic JS
  • isomorphic JS is going to likely require that we run NodeJS either in addition to PHP but more likely instead of PHP
  • Even if we are focused on admin tools, some of those admin tools run on user-facing pages: Toolbar, Quick Edit, Contextual Links, etc. Comments have also been mentioned.
  • User-facing admin tools could maybe be made more responsive with a client-side framework, but again... for quick loads, that means isomorphic and thus possibly NodeJS.
  • If we are running NodeJS on user-facing pages in order to speed up those admin tools, it may be difficult or impossible to do so alongside PHP rendering, which could force JS rendering of user-facing pages purely to improve admin tool responsiveness
  • As others have pointed out (perhaps more on the NodeJS thread), some of the admin tools may need to be changed pretty far back in the toolchain to run in JS. It's not impossible that we find that more and more of Drupal needs to be converted to JS to make admin tools more responsive
  • At some point it may seem silly to have half of Drupal built in JS and some in PHP. Thus, all of Drupal is converted to JS.

Maybe that's an extreme scenario, but I do not think it is implausible. There are huge technical implications and a huge amount of work that could be required simply because there is the notion that if we just use client-side frameworks for our admin, our UX will be better and front-end developers will like to work with Drupal more.

Drupal written entirely in JS I guess could still be called Drupal, but that's a pretty big change for a whole lot of people's lives that are predicated on their knowledge of PHP as a core part of their careers. I don't think it should be all that surprising, when such a big change is one possible result of this thread, that people want to make really, really, really sure that doing so is a good idea.

Crell’s picture

markabur: I think it's more accurate to say that being a "data serving machine" was a key goal of Drupal 8, and we made huge progress in that area but there's still plenty more to do. Also, there's no one-size-fits-all solution in that space.

And yes, using non-Drupal for a non-CMS project is totally legit. Making it easier to do so is one of the benefits of having a more conventional architecture now, so the relearning for backend devs is less. That doesn't mean having a more robust JS layer for the admin is a bad thing for a CMS; I agree that it can be a big help.

The problem does seem to be that the jump from "a more robust JS layer" to "let's pick one of the big frameworks and rewrite everything in it" was made, or implied, very prematurely.

tic2000’s picture

I follow this discussion since the start, but I said is not a decision I can influence so I will just wait to see a conclusion, if any.

But now I will give my 2 cents and I hope you will forgive me my rambling and my English which tend to degrade rapidly when I'm rambling.

A few years ago when I was told to implement some javascript on front end I always said "sorry, I can't" so my experience was importing a library, make drupal output the markup the library desire and implement the code required to initialize it which was copy paste.

Through the years that change and now I've built full sites back-end and front-end in javascript (node.js and angular 1).

Crell's presentation in Barcelona I think was on point. A lot of people are drawn to node.js for the "live" feelings they have. A user register to my site and if I'm on the users page I can see a new row appear in my table. I can send notification through a web socket and they appear live.

To achieve that using Drupal I would need to set a javascript which regularly calls the back-end asking "is there anything new?". And I need to query the users table to see if anyone new was added since last request. If it would work in a node.js way you could on hook_user_insert() send that request to everyone listening. So not only node.js doesn't need to "start-up" every time, but it doesn't need to query the table for everyone listening.

Regarding the SPA's, that's perfectly achievable in Drupal. a SPA still queries the back-end for each page and receives a json. You can use ajax on every link and button in drupal if you really want.

I think the focus should be on making that easier. If I want to request the page throug ajax to first have an option of receiving a full json, no rendering done, or json with stuff rendered. IE

{
sidebar_first:
  {blocks: [{
    title: "<h2 class=\"title\">Some Title</h2>",
    content: "<div class=\"bla-bla\">stupid block content</div>"
  }]},
main_region: {...}
}

You get the idea I hope. I would also like something similar for forms (note that I didn't look into D8 that much, so if it already exists ignore me).

Last year we built a site using D8 beta 4 and angular 1. I thought sending a node to be saved was overly complicated.

save: function (node, fn) {
    var data = {
        "_links": {
            "type": {"href": window.location.origin + "/api/rest/type/node/nodetype"}
        },
        "title": [
            {
                "value": node.title.value
            }
        ],
        "body": [
            {
                "value": node.body ? node.body.value : ""
            }
        ],
        "field_email": [
            {
                "value": node.field_email ? node.field_email.value : ""
            }
        ],
        "field_website": [
            {
                "uri": (node.field_website && node.field_website.value) ? "http://" + node.field_website.value.replace('http://', '') : ""
            }
        ]

    };

    var headers = {
        'Accept': 'application/json',
        'Content-Type': 'application/hal+json',
        'Accept-Language': 'en-us'
    }
    if (node.token) {
        headers['X-CSRF-Token'] = node.token;
    }

    $http({
        method: 'POST',
        url: '/api/entity/node',
        headers: headers,
        data: data
    }).success(fn)
      .error(fn);
}

Probably there was a reason for it, but why so complicated? For someone working with JS I would have expected something much simpler for the data. Why do I have to figure out that link ("/api/rest/type/node/nodetype")? Why not just send the request to /api/entity/node/nodetype or just declare the node type in the array along with the other values? Why the need to repeat "value" over and over, or uri? The back-end should know that field saves the data in a URI, or value or target_id. I would have liked my data to be something like:

var data = {
        type: "nodetype",
        "title": [
            node.title.value
        ],
        "body": [
            node.body.value
        ],
        "field_email": [
            node.field_email.value
        ],
        "field_website": [
            node.field_website.value
        ]
    };

Like others have said we should focus on changing some of the UI we have to use more ajax and see what issues we find and solve them. Then we can talk what framework (if any) we should use. Me personally I can get behind any of them even if I didn't use them yet because really is not that hard. Even in using node.js the more complex stuff was on back-end and the fact that there is nothing like Drupal or WordPress in node.js world makes this even more difficult.

Regarding the usage of Javascript. Angular 2 is in beta and I see of lot of tweets about yet another boilerplate, this is using X, Y and Z, while the other uses U, X and Z. On top of that every little bit of tutorial will end up with "you can get the code on github at ..." even if it's some basic filtering, or a todo list application. This doesn't happen with any other language.

catch’s picture

@webchick my point wasn't that we should do more usability testing (except when we have new things to test). The proposal does not at all mention those results and how a framework might or might not help fix those issues, but just mentioned facebook/twitter/google. As your list makes clear, some of our issues cannot be solved with js at all, some possibly could. As nod_ points out, @akalata's comment explains the concern better than either of our comments.

Also several people on this thread have already pointed out that the github stats are skewed towards fragmented projects (i.e. lots and lots of personal forks and duplication), and don't necessarily represent health at all. We really need to look at health/quality rather than pure numbers - both tumors and flowers 'grow'. If we moved Drupal.org project hosting to github and a 100% pull request workflow, that in itself might be enough to juke the stats in favour of PHP (i.e. assume one fork per patch contributor to each contrib project or core and you suddenly have tens or hundreds of thousands of forks after a year or two). It wouldn't represent a flowering of PHP at all though.

@mdrummond in case you've not seen it, a 'use node.js' discussion is happening at #2645666: [policy, no patch] Require Node.js for Drupal 9 core and rewrite some of Drupal's UI code from PHP to JS. As you point out that's one potential conclusion - all rendering happening in js, both client and server-side, and PHP only providing a REST API for that to interact with.

mdrummond’s picture

I think it is worth taking a step back to discuss the actual goals we want to achieve and the various ways we might want to achieve them outside of an issue is geared towards one possible solution for those goals.

So I opened a new issue: #2655556: [META] Dramatically improve Drupal-provided user interactions through perceived performance boosts and optimistic feedback

I hope this is taken in the spirit with which it is intended: a recognition of the importance of addressing the core issues at stake here.

I do not in way mean to diminish all of the good and important work that has been done to evaluate a variety of client-side frameworks. I do care about the goals at stake here, even if I have concerns about the suggested approaches. I'm hoping that a fresh start to the discussion can help reset things and calm the frustrations that many—including me—have felt. I'd humbly suggest that the discussion about what approaches we should look at might better fit with this new thread, while evaluations of the various client-side frameworks (even if we end up not using them) might better fit in this thread. That's just my opinion, though, so continue on as you please. :)

Crell’s picture

A recent episode of Full Stack Radio (podcast) (http://www.fullstackradio.com/32) had DHH (of Ruby on Rails fame) on to talk about Basecamp 3, and included some discussion of the front-end vs. back-end rendering question. His take-away: Rendering happens on the backend, duh, but use a websocket to send it back to the browser as quickly as possible.

The directly relevant portion is from about the 18-31 minute mark, but the whole episode is worth a listen. (He even has positive things to say about PHP! He's mellowed in recent years...)

twistor’s picture

stevector’s picture

Thanks Crell. That was a great podcast episode.

Some points that stood out to me:

  • The importance of using a technical architecture that matches the people building the software. DHH mainly discusses this concept in relation to team size (Microservices work well for very large teams or teams of teams).
  • That Turbo links go a long way toward fulfilling the experience of an uninterrupted application. #2603794: Add PJAX/Turbolinks support
  • The high cost of having duplicate template files as screens multiply: having different templates for client-side/server-side would be unmanageable for a project as large as Basecamp with their small team.
  • To build a product in a timely manner trade-offs must be accepted. "How do we get this $22,000 car to feel like a $32,000 car." To compete with products from companies with much larger budgets Basecamp must be much more deliberate about the trade-offs it makes.
  • If you have all the resources in the world you can write tests at every single level. DHH prefers having enough testing to make him confident in the product. "Testing against controllers and views at the same time gives you a lot of bang for the buck."
dawehner’s picture

If you look at the discussions there is a trend, that there is no clear discussion, because we are talking about various things at the same time.
Both improving the admin UX, the frontend UX, the developer experience, finding new contributors and expanding the entire community as
well as the search of a new fundamental base for Drupal itself.

You could have a much better discussion if we first discuss about personas of Drupal. Who are the people we want to help with:
Here are just a couple of questions, which don't define the personas at all. I just want to give an example of the range of users we have

* The individual who wants to setup a site
* The freelancer who want to build a site for their clients
* The agency who wants to work on big products

* Site users having an easy time to find good information on sites without technology barriers
* Content authors having a good time to enter information as best as possible
* Content authors having a good time to create the site as best as possible
* Marketers having the power to display what they think is important
* Site builders being able to achieve great things without a necessary technical background
* Ensuring a good life for the Drupal community?

If you look just at those examples, it seems to be that this touches the Mission and Principles
of Drupal. Maybe its just time to rediscuss that. Maybe we want to value the content authors over the minimal requirements? I strongly believe
that we first need to agree on that fundamentals, then go to personas and then maybe come back to the technical discussions we have at the moment.

For me this is not a technical problem at the moment: #2661180: Rediscuss the mission of Drupal

Wim Leers’s picture

#93: Thanks for that link! Note that Rails uses Turbolinks to make loading content/switching between pages as efficient as possible, without having to deal with lots of JavaScript. I plan to work on that in the next weeks/months, and I've already posted a screencast of a very rough PoC, in which it looks quite promising indeed. Thanks @twistor for already linking that: #2603794: Add PJAX/Turbolinks support.

I also listened to the podcast. I heard a few interesting statements not yet mentioned in #93 or #95, but especially relevant to this whole discussion.

  1. We consider server-side rendering as the baseline. Notifications are loaded via websockets. New pages are loaded via AJAX [because they use Turbolinks]. The next step (client-side MVC + JSON responses) is only taken if server-side rendering can't get fast enough: when there are fidelity issues. Example: datepicker, WYSIWYG.
  2. But what I found, is that there is a bunch of accepted wisdom: "You HAVE to use client-side MVC" — I actually have found the opposite: only the minority of the time you need that. Of course, a WYSIWYG editor that'd have to go the server for every keystroke, that wouldn't work. But for lots of other cases, such as menus, server side works awesome.
  3. Turbolinks is a layer that simulates in some cases the client-side MVC feel, or at least this notion of "the persistent process" [talking about the UX], without requiring you to do all this work on the client side.
  4. That's also how we use websockets — even individual [Basecamp] chat lines are rendered on the server side and sent down to the client. Which then means we can reuse those templates for all sorts of other things such as transcripts or search.

EDIT: cross-posted with #96.

attiks’s picture

prestonso’s picture

Status: Active » Postponed

Now that this issue has generated many other issues, and now that @Dries has made his position clear, I'm postponing this issue for the time being. We can reactivate when it becomes time. Thank you all for the wonderful discussion — this is what makes Drupal so special. We've planted a seed here that will yield great fruit.

Whether we roll out our own framework or go the Proudly Found Elsewhere route, I strongly believe Drupal will eventually need some kind of a system for client-side state management and rendering in order to match up to the competition. I spoke with @lauriii about this last week, and while Drupal should not become a framework-agnostic RESTful service (which wipes out site building functionality such as localization and layout management), we should evolve into a framework-agnostic front end by following the precedent of 5.x, in which developers could opt to choose their own template engine, but Drupal standardized on PHPTemplate by default. I expect the same to happen here (Smarty never took off due to PHPTemplate's convenience). In my view, this would require Drupal to export some information to JavaScript that can be leveraged by any client-side framework, similar to what already happens to some degree with drupalSettings.

Two weekends ago, a group of Drupalists gathered at DrupalCamp New Jersey 2016 to discuss progressive decoupling and the future of Drupal's front end. I recorded the conversation and transcribed it as well if you prefer reading what was said. I'm looking forward to continuing this conversation at DrupalCons and camps in the future.

yoroy’s picture

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

J. Cohen’s picture

Vue.js is the 3rd most popular JS framework on Github and might be #2 or #1 by the end of 2017. It's worth considering, if it isn't too late.
https://github.com/showcases/front-end-javascript-frameworks

If a Turbolinks feature gets added to Drupal core, please keep it off by default. Turbolinks are really unpleasant from a user standpoint (at least this user), making the normal loading process appear very slow. I think it's because the width of the progress bar is the entire screen, giving the impression that the loading process has a long "distance" to go.

I cringe every time I see those that turbolinks loading bar (including YouTube and Github). If Drupal adds turbolinks to core and puts it on by default (or recommends it), it will make Web surfing more painful.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

bdichgans’s picture

Just want to throw in glimmer.js as an option. It is just the View layer of Ember.js and could be a good way to dive into the frontend framework world.