[plan] Modernize Drupal's JavaScript

Problem/Motivation

Back in 2007, the very first iPhone was released, the last Harry Potter book was published, and Britney Spears had an extra tough year. Also, Drupal 5 shipped with the jQuery JavaScript library.

Over ten years later, while Drupal's JavaScript has seen some great advancement since those days—for example, adopting AirBnB standards, and introducing a JavaScript build process, and upgrading to the most recent versions of jQuery and jQuery UI in 8.5—there still remains a lot of legacy JavaScript code in core from "back in the day" that makes it sub-optimal to maintain.

Additionally, modern JavaScript frameworks such as React, Angular, and Vue have gained a lot of popularity as tool kits that remove a lot of the pain around building rich, dynamic, "app-like" interactions. Drupal's current administrative UI feels very much like something that was invented in the mid-00s and has been stuck there since: basic and plain-looking forms, slow server round-trips required to make any changes, the lack of expected visual niceties from other solutions, such as live previews and drag and drop.

Therefore, our plan of attack is three-fold:

1. Simplify. Craft a redesigned, modern administrative experience for Drupal with the aim to make it more approachable for site builders and content authors, using a JavaScript framework (React) to build it out as a single-page application. In so doing, "dog food" Drupal's REST API, which will lead to a much better developer experience for all decoupled builds, regardless of chosen framework. The intent is to create a standalone decoupled application, *not* to change anything about the Drupal theme and render layers.
2. Modernize. Work on efforts around Drupal core's JavaScript developer experience to bring it into the 21st century. For example, modernizing the code base to ensure consistency and adherence with best practices, adding native tools for testing JavaScript with JavaScript. Our aim here is to help build a community of JavaScript-savvy Drupalist core developers and happy, energized JavaScript maintainers.
3. Embrace. Throughout, embrace the workflows, tools, and practices familiar to existing JavaScript developers, with the aim to bring more of these developers into the Drupal community, both as users and as contributors. This means primarily developing and reviewing code related to the admin UI on GitHub (where the JavaScript community is centralized), using standard tools like create-react-app which JS developers are already familiar with, and moving code and discussion to Drupal.org only when chunks of work are ready for review. (For issues that impact "upstream" Drupal core code, those discussions will take place on Drupal.org.)

Proposed resolution

This initiative will focus on two main areas - reducing the maintenance pain for our current codebase (as this will be with us for some time still), and creating a new, alternative admin experience for Drupal.

1. Remove architectural blockers to decoupled builds by dogfooding Drupal’s API in Drupal.
2. Create an alternative admin experience for Drupal as a single-page React application.
3. Create a JavaScript-based automated test suite for JavaScript
4. Unify (and automate) JavaScript coding standards

New Admin Experience

Remove architectural blockers to decoupled builds (Target: 8.6)
While Drupal has supported "headless" builds since even the good ol' vintage Flash days of the '00s, there's always more that could be done to better support decoupled front-ends and make Drupal more enjoyable to work with for JavaScript developers. This sub-initiative covers these items.

These issues are places we need additional input to help develop a firm implementation plan on each of the following areas. These issues generally benefit not only React, but other frameworks as well.

Work with the API-First team to accomplish:

• Get JSON API in core (covers entities) #2836165: New experimental module: JSON API
• Determine method to expose non-entity endpoints (e.g. modules, permissions), as well as simple configuration. #2913790: List of missing APIs
• Fix limitations in upstream REST API, such as #1927648: Allow creation of file entities from binary data via REST requests and #2300677: JSON:API POST/PATCH support for fully validatable config entities

Create an alternative admin experience for Drupal as a single-page React application (Target: 8.7)

Work for this part of the initiative is happening at https://github.com/jsdrupal/drupal-admin-ui

• Designs and specs for the redesigned admin experience: #2902399: Redesign the Admin UI
• Prototype phase: Work together with UX team to create a better experience around the permissions page as an initial MVP (there is work currently in progress to build out a skeleton version of the permissions page to figure out the architecture https://github.com/jsdrupal/drupal-admin-ui/issues/7)
• Progressively build out additional pages, starting with self-contained forms, then to basic configuration forms, then onto very complex pages.
• As we build pages, extract reusable pieces into a generic component library for both general and Drupal-specific UI patterns, e.g. dropbutton, table select, drag and drop, file upload widget, etc. in the Seven style guide.

Existing Codebase

These set of issues tackle improvements to the existing code base, to make our JavaScript more approachable to new developers and more maintainable for core.

Create a JavaScript-based automated test suite for JavaScript (Target: 8.6)
While the PHP side of our code base is kickin' strong thanks to tools like PHPUnit, we're currently using PHP to also test our JavaScript code, which is suboptimal. (Testing is hard enough as it is, and JavaScript developers would vastly prefer to write tests in their "native tongue." :))

This sub-initiative aims to replace phantomjs (which is no longer needed nor maintained), with native headless Chrome provided by nightwatch. Completing this work is essential to ensuring we do not introduce regressions as we proceed with the other items in the roadmap.

Work is tracked under the JavaScriptTest issue tag.

• Modernizing our automated test bot infrastructure to support JavaScript:
  • #2869825: Leverage JS for JS testing (using nightwatch)
  • #2775653: JavascriptTests with webDriver
  • #2894501: Allow for separate mink driver settings for Javascript tests
• Filling in missing test coverage, such as:
  • #2892872: [META] Add Javascript tests for all Form API's #states
  • #2754985: [backport] Add JavaScript test coverage for adding an exposed filter in Views UI
  • In general, the things listed at 8.x JS manual test plan (which needs to be updated to include some of the more recent Drupal 8 features).

Unify (and automate) JavaScript coding standards. (Target: 8.6)

This sub-initiative is about documenting and enforcing a consistent set of coding standards to help make JavaScript code in Drupal easier to approach, debug, and maintain. We use the ESLint utility to enforce JavaScript coding standards in our code base.

Note: this is a great place for contributors who are new to the core JavaScript initiative to get their start!

• #2912962: Step 1 JS codestyle: [meta] Fix JS coding standards in core

Get involved!

Interested in teaming up? Come meet with us in #javascript on Drupal Slack. In addition to general chat, we hold a weekly text-only meeting on Mondays at 17:30 UTC (what is that in my timezone?) Follow @jsdrupal on Twitter for updates, and you can find an overview of our work and its current status at https://trello.com/b/P06ukajf/drupal-javascript-initiative (we’ll be moving this to a GitHub project board in the near future).

We especially are in need of folks to help review/unblock issues tagged with "JavaScript", particularly in any of the above areas.

Addendum (things we aren’t focusing on right now, but are keeping in mind):
https://github.com/jsdrupal/drupal-admin-ui/issues?q=is%3Aopen+is%3Aissu...
#2920196: Which capabilities should modules have in a new JS based admin UI: Discusses whether or not we maintain support for things like form alters, themeable content, and so on, in a decoupled world.
#2913372: Allow forms to be defined in three segments: schema, UI, data: Essential to any form rendering to the client side.
Providing auto-generation of simple config, content + config entity forms based on config schema data: #2949888: Enhance config schema for richer default experiences