Problem/Motivation

We want Drupal 8 to be fast by default. One aspect of being fast by default is the front-end performance, and one aspect of that is the amount of data we send to the browser. JavaScript minification can be a huge help in that regard, but has long been impossible, for multiple reasons:

  1. The most fundamental blocker. Drupal <8 used to be developer-friendly by default rather than fast by default, which meant that if we'd ship with JS aggregation/minification enabled by default, we'd have to detect file system changes on every request, which would be detrimental for performance. #2226761: Change all default settings and config to fast/safe production values changes that: aggregation can be enabled by default, and when developing, it's easy to disable aggregation.
  2. The secondary blocker. The license problem: proper minification deletes all non-essential data, including license information. That's the part this issue addresses, in part. This issue is about adding the necessary metadata (because not every JS file contains license information, we cannot rely on that), which requires an API change.
  3. Exposing that license information: the closing stone for the license problem: #2258313: Add license information to aggregated assets.
  4. The last mile. We couldn't find consensus on which JS minification tool to use. This issue doesn't aim to solve that, it only aims to make it possible to add JS minification at a later point in time to Drupal 8, since adding that is not an API change, it's improving an existing feature. (Our current "minification" strategy is: "just use the entire file, and append a defensive semi-colon".) That could even happen after beta 1 (see #3).

Proposed resolution

Required reading:

In essence, JSLWL requires us to list every JS file on the site, with its license and full source.

So, the proposed solution takes these steps:

  1. In Drupal 8, all assets must be declared through libraries. Libraries are declared in <module name>.libraries.yml files. We add a new license key with a map containing the following key-value pairs: name (license name), remote (URL to the project's license) and gpl-compatible (whether this license is GPL-compatible or not). e.g.:
    backbone:
      remote: https://github.com/jashkenas/backbone
      version: 1.1.0
      license:
        name: MIT
        remote: https://github.com/jashkenas/backbone/blob/1.1.0/LICENSE
        gpl-compatible: true
      js:
        assets/vendor/backbone/backbone.js: { weight: -19 }
      dependencies:
        - core/underscore
    
  2. Libraries without the license key are assumed to be "GPL2+", like Drupal, and are hence GPL-compatible. This means that none of the modules on drupal.org that include JavaScript code need to define licenses, unless they reference externally hosted JavaScript, which is a tiny minority.
  3. This allows us to generate a "JavaScript License Information" page, as mandated by JSLWL. We will implement that in #2258313: Add license information to aggregated assets.
    (Why not in this issue? Because it doesn't require API changes and contains some contentious things, whereas adding license metadata is not contentious.)

Remaining tasks

Review.

User interface changes

None.

API changes

  1. License information in *.libraries.yml files.
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Wim Leers’s picture

Status: Active » Needs review
FileSize
19.89 KB
corbacho’s picture

If test passes, this patch is ready IMHO. The license links were reviewed in the parent issue already:
https://drupal.org/node/2258313#comment-8749623

Status: Needs review » Needs work

The last submitted patch, 1: asset_libraries_licenses-2276219-1.patch, failed testing.

corbacho’s picture

Patch is missing the file library_test_files/licenses.libraries.yml ?

sun’s picture

  1. <code>
    +++ b/core/core.libraries.yml
    @@ -277,18 +293,30 @@ drupal.vertical-tabs:
    +    gpl-compatible: true
    

    Still wondering whether we can find a shorter name for this key...

    Since "gpl-compatible" is not fully accurate anyway (as it would have to be "gplv2-compatible"), perhaps we can shorten it to just "compatible"?

  2. <code>
    +++ b/core/core.libraries.yml
    @@ -658,6 +743,10 @@ matchmedia:
    +    url: https://github.com/paulirish/matchMedia.js/blob/0.1.0/LICENSE.txt
    
    @@ -672,6 +765,10 @@ modernizr:
    +    url: https://github.com/necolas/normalize.css/blob/master/LICENSE.md
    

    The specified URLs are inconsistent — some are linking to the latest version in the mainline/master branch, some others are linking to the exact version of the included library.

    While it would probably be accurate to link to the version-specific license, I fear that people will forget to update this URL when updating a library...

  3. <code>
    +++ b/core/lib/Drupal/Core/Asset/LibraryDiscovery.php
    @@ -174,6 +174,15 @@ protected function buildLibrariesByExtension($extension) {
    +      // Assign Drupal's license to libraries that don't have license info.
    +      if (!isset($library['license'])) {
    

    Based on an earlier discussion in the previous issue, I think this logic should be hardened:

    1. If a library specifies a 'remote', then it MUST specify a 'license'. (should be enhanced to also require 'version' in a separate issue)

    2. If a library does not specify a 'remote' and 'license', then the default license is injected.

  4. <code>
    +++ b/core/modules/system/system.routing.yml
    @@ -408,3 +408,11 @@ system.batch_page.json:
    +system.javascript_license_web_labels:
    

    Doesn't appear to belong to this patch?

  5. <code>
    +++ b/core/tests/Drupal/Tests/Core/Asset/LibraryDiscoveryTest.php
    @@ -376,6 +376,88 @@ public function testLibraryWithDataTypes() {
    +    $this->assertEquals($library['license'], $expected_license, 'Expected license metadata found.');
    ...
    +    $this->assertEquals($library['license'], $expected_license, 'Expected license metadata found.');
    

    Please remove the custom assertion messages from all of these.

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
24.76 KB
9.91 KB
8.18 KB

#4: D'oh :( Fixed.

#5:

  1. compatible is so vague. Let's just rename it to gplv2-compatible? :)
  2. The specified URLs are inconsistent — some are linking to the latest version in the mainline/master branch, some others are linking to the exact version of the included library.

    Actually, they are consistent. Whenever a specific tag is available, it points to that. Otherwise, it points to the best alternative.

    While it would probably be accurate to link to the version-specific license, I fear that people will forget to update this URL when updating a library...

    That's not a new problem. People also tend to forget to update the version key.

  3. Agreed, done.
  4. Oops, failed to clean that up. Removed.
  5. Oh, heh, sure :) Done.
Wim Leers’s picture

What else is left here? Let's get this in!

Wim Leers’s picture

FileSize
23.66 KB

Reroll; this was broken by #2223143: Consolidate library extension caches into a single cache entry.

Also verified that no 3rd party libraries have been added in the mean time; so this patch should once again be good to go.

dawehner’s picture

Yeah to unit tests!

+++ b/core/tests/Drupal/Tests/Core/Asset/LibraryDiscoveryParserTest.php
@@ -346,6 +346,123 @@ public function testLibraryWithDataTypes() {
+   * @expectedException \Drupal\Core\Asset\Exception\LibraryDefinitionMissingLicenseException
+   * @expectedExceptionMessage Missing license information in library definition for 'no-license-info-but-remote' in core/tests/Drupal/Tests/Core/Asset/library_test_files/licenses_missing_information.libraries.yml: it has a remote, but no license.
+   *
+   * @covers ::buildLibrariesByExtension()
+   */
+  public function testLibraryWithMissingLicense() {
...
+    $libraries = $this->libraryDiscoveryParser->buildByExtension('licenses_missing_information');
+    $library = $libraries['no-license-info-but-remote'];
+    $this->assertCount(1, $library['css']);
+    $this->assertCount(1, $library['js']);
+    $this->assertTrue(isset($library['license']));
+    $default_license = array(
+      'name' => 'GNU-GPL-2.0-or-later',
+      'url' => 'https://drupal.org/licensing/faq',
+      'gpl-compatible' => TRUE,
+    );
+    $this->assertEquals($library['license'], $default_license);

Doesn't all the code not run if buildByExtension throws an exception?

Wim Leers’s picture

FileSize
23.16 KB
1.71 KB

#9: good catch; you're absolutely right! I think that might be a remnant from an early iteration. Removed.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

Thank you!

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed 256b197 and pushed to 8.x. Thanks!

  • alexpott committed 256b197 on 8.x
    Issue #2276219 by Wim Leers: Asset libraries should declare their...
nod_’s picture

Doesn't this need a change record?

alexpott’s picture

Status: Fixed » Active
Issue tags: +Needs change record

Good point - it does. I thought we might get away with updating https://www.drupal.org/node/2201089 but I don't think so

pbull’s picture

PHPUnit tests are failing with:

Trying to @cover or @use not existing method "\Drupal\Core\Asset\LibraryDiscoveryParser::buildLibrariesByExtension".

Should the annotations in core/tests/Drupal/Tests/Core/Asset/LibraryDiscoveryParserTest.php now be using @covers ::buildByExtension() instead of @covers ::buildLibrariesByExtension()?

Mile23’s picture

Status: Active » Needs review
FileSize
2.7 KB

Here's a patch to fix the errors @pbull mentions in #16.

Fixes some @covers annotation, and also test method names to reflect the method names being tested.

There really should be a flag in the phpunit.xml.dist file to check for bad coverage annotation within continuous integration. Discussion here: #2105583: Add some sane strictness to phpunit tests to catch risky tests

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

These tests don't fail, unless you run them in static mode or code coverage, just to clarify.

alexpott’s picture

Status: Reviewed & tested by the community » Active

Okay, committed 8d6f823 and pushed to 8.x. Thanks!

Can we get a change record now?

  • alexpott committed 8d6f823 on 8.x
    Issue #2276219 followup by Mile23: Asset libraries should declare their...
alexpott’s picture

Status: Active » Needs work
alexpott’s picture

Issue tags: +Missing change record
Wim Leers’s picture

Status: Needs work » Fixed
Issue tags: -sprint, -Needs change record, -Missing change record

@Mile23: thanks for #17!

Change record published: https://www.drupal.org/node/2307387.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

dawehner’s picture

YesCT’s picture

Issue tags: -front-end performance +frontend performance

changing to use the more common tag, so the less common one can be deleted, so it does not show up in the auto complete and confuse people.