Use package.json and rollup to manage third party JS libraries

Problem/Motivation

Keeping libraries up to date is a manual process. Need to visit each library and check if there was a new release.

Proposed resolution

Introduce an automated way to check library status and update dependencies.

Workflow

The proposed workflow to update a library is:

yarn outdated
yarn upgrade X Y Z
yarn run rollup # command name subject to change.

Details

The goal is to provide a predictable experience on usage, minification, and debugging.

Overview

Rollup is used because it's easy to use and there is not much configuration necessary. It's also suited to package individual files/libraries whereas webpack is more turned towards handling a single project as a whole. Because Drupal is using it's own library/dependency handling it's easier to have individual files that are aggregated later.

To manage a library there are 4 steps:

1. yarn add -D <library> Add the package as a dev dependency with yarn, if a package doesn't exists in npm, use the git repository directly (needed for farbtastic)
2. Configure the rollup file by declaring 2 steps:
   1. Step 1: is to bring the unimified version of the library into the assets/vendor/<library>/<library>.js file. This is done in rollup with a module import, rollup has a way of declaring 'virtual' module, that can declared inline. In short we can do the equivalent of include "file"; with
      

      {
        input: "entry",
        plugins: [
          // avoid the need for a dedicated file somewhere on disk.
          virtual({ entry: 'import "<library>";' }),
          // Does the actual code 'include'
          resolve(),
        ],
        output: { file: 'assets/vendor/<library>/<library>.js },
      }
        

      Now depending on the library format for umd files a simple import "<library>" is needed, when the library is a proper JS module we need to be more explicit with import Lib from "<library>"; export default Lib; so that the second step works (this is the case for sortable, js-cookies, once).

   2. Step 2: is process (transform is needed + minifiy) the previous file and generate the corresponding map file. The transform is to serve the library in a format appropiate for ES5 browsers (umd or iife). If the file was already in a umd format, there is little to do (just need to change some terser configuration to prevent ending up with additional global variables). If the library was in a JS module format, it is transformed into a iife (not a umd because it's smaller and I haven't seen that format really used for anything on the browser side lately, trivial to change though.). An example of the code transformation happening for JS module to iife.
      

      // 1. A JS module in a 'api.js' file
      const a = true;
      const api = () => a;
      export default api;  
       

      // 2. contents of 'api.js' transformed into an iife
      var api = function () {
        const a = true;
        const api = () => a;
        
        return api;
      }();
       

      // 3. contents of the 'api.js' iife is minified with terser
      var api=function(){"use strict";return()=>true}();
      //# sourceMappingURL=api.min.js.map
      
        

      in the rollup file it's presented like this:

      {
        input: 'assets/vendor/<library>/<library>.js',
        // minify code
        plugins: [terser()],
        output: {
          file: 'assets/vendor/<library>/<library>.min.js',
          // Generate a sourcemap for debuggability.
          sourcemap: true,
          // Do not copy the unminifed source into the map file, we already have it in the <library>.js file.
          sourcemapExcludeSources: true,
          format: 'iife', // needed only for JS modules
          name: 'api',  // needed only for JS modules
        },
      }
      
      

3. Declare the library in core.libraries.yml
4. Profit

Because changing dependencies file names can be disruptive there are some inconsistencies (underscore-min.js instead of underscore.min.js, farbtastic.js having the minified version of the library, etc.).

Minification/Debugging

The reason we don't use the minified version of the library is to make it possible to offer a consistent debugging experience.

Problem

In core map files are not always available and some behave 'strangely'. Because we manually copy/pasted library code in an arbitrary folder that doesn't follow the library package structure some strange things can happen. Let's take jquery-form as an example:

// Extract of the structure for the jquery-form package.
package.json
dist/jquery.form.min.js
dist/jquery.form.min.js.map
src/jquery.form.js

// Folder structure in drupal/core directory
assets/vendor/jquery-form/jquery.form.min.js
assets/vendor/jquery-form/jquery.form.min.js.map

With the map file pointing to ../src/jquery.form.js it gives us an experience where the jquery.form.js file appears to be under assets/vendor/src/jquery.form.js (it is even worst for the popper lib which appears directly under assets/src!). There are also libraries with map files that are not referenced, and not shown in the dev tools, which makes them useless.

Solution

When we generate the map file everything behaves as expected:

Remaining tasks

• Feedback
• Dependency evaluation
• File size comparison

User interface changes

None

Release notes snippet