Problem/Motivation

We want Drupal 8 to be fast by default. One aspect of being fast by default is the front-end performance, and one aspect of that is the amount of data we send to the browser. JavaScript minification can be a huge help in that regard, but has long been impossible, for multiple reasons:

  1. The most fundamental blocker. Drupal <8 used to be developer-friendly by default rather than fast by default, which meant that if we'd ship with JS aggregation/minification enabled by default, we'd have to detect file system changes on every request, which would be detrimental for performance. #2226761: Change all default settings and config to fast/safe production values changes that: aggregation can be enabled by default, and when developing, it's easy to disable aggregation.
  2. The secondary blocker. Far too often, we don't know and cannot determine the license of an asset! #2276219: Asset libraries should declare their license addresses that.
  3. The tertiary blocker. The license problem: proper minification deletes all non-essential data, including license information. That's the part this issue addresses.
  4. The last mile. We couldn't find consensus on which JS minification tool to use. This issue doesn't aim to solve that, it only aims to make it possible to add JS minification at a later point in time to Drupal 8, since adding that is not an API change, it's improving an existing feature. (Our current "minification" strategy is: "just use the entire file, and append a defensive semi-colon".) That could even happen after beta 1 (see #3).

Proposed resolution

Required reading:

In essence, JSLWL requires us to list every JS file on the site, with its license and full source.

So, the proposed solution takes these steps:

  1. #2276219: Asset libraries should declare their license associates the license information with each asset library.
  2. This allows us to generate a "JavaScript License Information" page, as mandated by JSLWL. We link to that page on every Drupal page that contains any JS. And on that page, we can list every JS asset along with its license, thanks to 1.
  3. However, we must also list the license and source for aggregated JavaScript assets (which we want to be minified in the future). JSLWL allows us roughly 2 ways of doing that:
    The source code file can be a single, unminified JavaScript file, a .tar.gz archive, or a .zip archive. If a source archive includes multiple JavaScript files, the archive must include a file named 00-INDEX that lists the order in which individiual source files should be concatenated to produce a single file that's equivalent to what's hosted on the site.

    So either: an unminified aggregate, or an archive containing all aggregated JS files and a describing file. The latter is much more painful to do, so we choose the first, for which we already have the necessary infrastructure. The JS asset collection optimizer (asset.js.collection_optimizer) tracks its aggregated (and minified) JS files (we can get them via AssetCollectionOptimizerInterface::getAll()); by adding a secondary JS asset collection optimizer asset.js.collection_optimizer_license_web_labels_annotator which annotates each contained asset (with a link to the license information) without minifying it, we effectively get two key-value maps with the same keys (because the same collections of JS assets are passed to them and they use the same JS collection grouper) but with different files (one minified, another annotated and unminified). On the JSLWL "JavaScript License Information" page, we can now list the unminified annotated aggregate as the source for the minified aggregate.

Remaining tasks

Review.

User interface changes

None that are visible, but a new "JavaScript License Information" page at /system/jslicense.

API changes

  1. (Not really an API change, but an internal behavior change.) An unminified aggregate is generated for every minified aggregate.
  2. Route/response addition: /system/jslicense.
  3. (Not really an API change, but a markup addition.) Each page has a <link rel="jslicense" href="/system/jslicense" /> tag in the HTML head.
CommentFileSizeAuthor
#96 interdiff_91-96.txt5.72 KBravi.shankar
#96 2258313-96.patch29.32 KBravi.shankar
#91 interdiff-89-91.txt22.82 KBnod_
#91 core-jswl-2258313-91.patch29.27 KBnod_
#89 interdiff-86-89.txt3.19 KBnod_
#89 core-jswl-2258313-89.patch14.51 KBnod_
#84 core-jswl-2258313-84.patch3.14 KBnod_
#80 core-jswl-2258313-80.patch3.93 KBnod_
#79 core-jswl-2258313-79.patch4.07 KBnod_
#76 interdiff_72-76.txt9.08 KBravi.shankar
#76 2258313-76.patch14.07 KBravi.shankar
#72 reroll_diff_55-72.txt6.27 KBravi.shankar
#72 2258313-72.patch14.37 KBravi.shankar
#55 drupal-n2258313-55.patch14.48 KBdamienmckenna
#55 drupal-n2258313-template_preprocess_html_beta11.txt4.86 KBdamienmckenna
#55 drupal-n2258313-theme.txt1.18 KBdamienmckenna
#51 implement_js_web-2258313-51.patch16.01 KBlauriii
#40 js_license_web_labels-2258313-40.patch15.5 KBlauriii
#36 js_license_web_labels-2258313-36.patch16.67 KBwim leers
#31 interdiff.txt1.43 KBwim leers
#31 js_license_web_labels-2258313-31.patch16.67 KBwim leers
#27 interdiff.txt15.03 KBwim leers
#27 js_license_web_labels-2258313-27.patch16.89 KBwim leers
#23 js_license_web_labels-2258313-23-DO_NOT_REVIEW-do-not-test.patch15.22 KBwim leers
#21 js_license_web_labels-2258313-21-DO_NOT_REVIEW-do-not-test.patch37 KBwim leers
#12 interdiff.txt2.29 KBwim leers
#12 js_license_web_labels-2258313-12.patch37.22 KBwim leers
#10 interdiff.txt8.61 KBwim leers
#10 js_license_web_labels-2258313-10.patch37.02 KBwim leers
#6 interdiff.txt1.46 KBwim leers
#6 js_license_web_labels-2258313-6.patch37.09 KBwim leers
#1 js_license_web_labels-2258313-1.patch36.4 KBwim leers
#86 core-jswl-2258313-86.patch10.9 KBnod_
#86 interdiff-84-86.txt7.37 KBnod_

Comments

wim leers’s picture

Status: Active » Needs review
StatusFileSize
new36.4 KB
wim leers’s picture

Issue summary: View changes

To clarify: the goal is for this issue to allow us to add JS minification at a later point in the Drupal 8 cycle. That could be after beta 1 (but before RC) or even Drupal 8.1.0 or 8.2.0.

wim leers’s picture

#2226761: Change all default settings and config to fast/safe production values landed! If we can get this in also, we can add JS minification to D8 at a later time.

corbacho’s picture

Status: Needs review » Needs work

I checked every License link and they are valid working links with license info.

  1. +++ b/core/core.libraries.yml
    @@ -279,18 +295,30 @@ drupal.vertical-tabs:
    +    name: GPL2
    

    GPL2 should be "GNU-GPL-2.0-or-later"

  2. +++ b/core/core.libraries.yml
    @@ -342,7 +386,11 @@ jquery.once:
    -  version: &jquery_ui 1.10.2
    

    At least to me looks more clear with "jquery_ui_version" than "jquery_ui"

  3. +++ b/core/core.libraries.yml
    @@ -667,6 +752,10 @@ matchmedia:
    +    remote: https://github.com/Modernizr/Modernizr/blob/v2.6.2/readme.md
    

    Maybe direct link? http://www.modernizr.com/license/

  4. +++ b/core/core.libraries.yml
    @@ -674,6 +763,10 @@ modernizr:
    @@ -691,6 +784,9 @@ picturefill:
    

    Missing license info for picturefill and matchmedia ?

  5. +++ b/core/includes/theme.inc
    @@ -2082,6 +2082,14 @@ function template_preprocess_html(&$variables) {
    +      '#markup' => l(t('JavaScript license information'), 'system/jslicense', array('attributes' => array('rel' => 'jslicense', 'class' => 'hidden'))),
    

    A link in every page? This is excessive IMHO, but at least is hidden (although the rules are: (copy/paste) "This link can be small, but it should be clearly visible to people who visit your site.".

  6. +++ b/core/lib/Drupal/Core/Asset/LibraryDiscovery.php
    @@ -249,6 +258,13 @@ protected function buildLibrariesByExtension($extension) {
    +          if ($type === 'js' && !$library['license']['gpl-compatible']) {
    

    This is excessive IMHO. Does this means that every js library that does not have the 'gpl-compatible' tag will affect negatively the performance of the site? What about non-external libraries, like drupal-ajax ? Are they bundled together?

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new37.09 KB
new1.46 KB

Thanks for the review!

  1. Changed.
  2. Agreed, that's why I made the change. You're just +1ing this renamed YAML alias, right?
  3. Thanks, changed.
  4. I figured that made most sense because we currently have modified versions of those two libraries. But yes, let's add the license information now. Fixed.
  5. This is what the FSF, EFF etc. sites do, and what the standard prescribes. There was talk of using a <link> element instead. Though that's not yet in the standard, IMHO we could do that too, because it enables the same thing: automated discovery of JS asset licenses.
  6. drupal.ajax is GPL-compatible, so of course it's aggregated. Everyt JS asset used in Drupal core is GPL-compatible, and almost every JS asset in modules or themes on drupal.org is. This is necessary to be able to describe the license of aggregated JS assets.
    I could go one step further, and allow non-GPL-compatible assets also to be aggregated together, but in a separate group, so that we'd still be able to determine the license of JS aggregates.
corbacho’s picture

2. Indeed +1 the renamed the YAML alias to "jquery_ui_version". (I understood the opposite by mistake)
And, yes, I really like this patch and the motivation behind, in the meanwhile doesn't affect negatively the performance. Thank you for working on this.

moshe weitzman’s picture

Looks good. Two minor questions ...

  1. +++ b/core/includes/theme.inc
    @@ -2059,6 +2059,14 @@ function template_preprocess_html(&$variables) {
    +
    +  // On each page that uses JavaScript, add the mandatory JavaScript Web License
    +  // Labels link.
    +  if (count(drupal_get_js('header')) || count(drupal_get_js('footer'))) {
    +    $variables['page_bottom'][] = array(
    +      '#markup' => l(t('JavaScript license information'), 'system/jslicense', array('attributes' => array('rel' => 'jslicense', 'class' => 'hidden'))),
    +    );
    +  }
    

    A render array is better added in hook_page_build() IMO but it is a minor distinction.

  2. +++ b/core/modules/system/lib/Drupal/system/Tests/Common/JavaScriptTest.php
    @@ -38,6 +38,11 @@ public static function getInfo() {
    +
    +    // Ensure the system.javascript_license_web_labels route is available.
    +    $this->installSchema('system', array('router'));
    +    \Drupal::service('router.builder')->rebuild();
    +
    

    Why is this needed? Do we really start tests without a router table?

sun’s picture

  1. +    name: MIT
    +    name: Public Domain
    +    name: GNU-GPL-2.0-or-later
    

    Hm. This adds many custom license names, for which no schema/definition/documentation seems to exist.

    For example, you've added "Public domain", but that doesn't exist on http://www.gnu.org/licenses/javascript-labels.html

    At minimum, I think we need to add a license name validation to the processing code (throwing an exception when encountering an unknown license name).

  2. +    remote: https://github.com/jashkenas/backbone/blob/1.1.0/LICENSE
    

    The existing 'remote' key name refers to a "git remote repository".

    This isn't a remote, it's just a 'url'.

    I also wondered whether it shouldn't be 'source', which would be in line with http://www.gnu.org/philosophy/javascript-trap.html#AppendixA, but I guess 'source' might be confusing...

  3. +    gpl-compatible: true
    

    I wonder whether this key is really necessary?

    Why can't we derive that flag automatically from the license name?

  4. +    name: GNU-GPL-2.0-or-later
    +    remote: http://malsup.github.com/gpl-license-v2.txt
    +    gpl-compatible: true
    

    The issue summary states that GPLv2+ is the default, because it's Drupal's overall license, so why are we specifying it?

  5. +    name: GNU-GPL-2.0-or-later
    

    All of composer.json, package.json, and bower.json have a license key in their schema.

    I wonder whether we shouldn't use the license names of those much more common schemata instead and translate them into JSLWL identifiers?

wim leers’s picture

StatusFileSize
new37.02 KB
new8.61 KB

Thanks for the reviews!

#8:

  1. Agreed, and that's what I did first. But that causes slightly different markup to be output: it causes a "region" wrapper <div> to be created, which is not the case with this approach. Since it's a hidden link anyway, I think it's better to not create the "region" <div>.
  2. Because it's a DUTB test.

#9:

  1. The license names listed at http://www.gnu.org/licenses/javascript-labels.html are not mandatory, they're considered "good" ( Good license identifiers and their associated links are: […]). Obviously, the list they have there is not even close to comprehensive.
  2. Aha! Yeah, I wanted to keep consistency, I never realized "remote" stood for "remote git repo". Thanks for that, I agree that "url" is much better :) I like your thinking regarding "source", but I agree it'd be confusing. So I went with "url".
  3. Because there are many GPL-compatible licenses, and aside from that, there are numerous licenses. We can't possibly include a comprehensive list to do this automatically.
  4. Good point. I figured we wanted to explicitly define the licenses of vendor libraries, because we (Drupal) don't own them. That removes any ambiguity. I don't have a strong opinion though, so if you want me to change that, I'll do it.
  5. Again, JSLWL doesn't have enforced "license identifiers"; it's free-form. Easily demonstrated by looking at https://weblabels.fsf.org/www.fsf.org/CURRENT/, where they don't use "GNU-GPL-2.0-or-later", but "GNU General Public License version 2 or later".

P.S.: wow, I didn't realize "schemata" was an alternative plural form of "schema"! :D

corbacho’s picture

Status: Needs review » Needs work
  1. +++ b/core/lib/Drupal/Core/Asset/JsCollectionOptimizer.php
    @@ -81,11 +91,13 @@ public function optimize(array $js_assets) {
         // Now optimize (concatenate, not minify) and dump each asset group, unless
         // that was already done, in which case it should appear in
    -    // system.js_cache_files.
    +    // system.js_optimized_aggregate_files (but also the unoptimized equivalent
    +    // in system.js_unoptimized_aggregate_files, for JavaScript Web License
    +    // Labels compliance).
    

    The added comment doesn't read very easily, or is it me?. There is no verb. Changing "but also" to "and also" sounds better?

  2. +++ b/core/modules/system/lib/Drupal/system/Controller/AssetLicenseInfoController.php
    @@ -0,0 +1,158 @@
    +        $license_column = l($library['license']['name'], $library['license']['remote']);
    

    still 1 more "remote" left

About sun's 4. By using license 'name' in core, we give example to contrib. Also, what if Drupal changes to GPL3 at some point? But no strong opinion here.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new37.22 KB
new2.29 KB
  1. Agreed; rewritten that comment. Should be much clearer now.
  2. Well-spotted, thanks!

RE: GPL3: that won't be a problem, since Drupal is already marked as "GPL2+". The only difference it will make here is that our aggregates would be labeled GPL3, and we could also include Apache-licensed JS assets in our JS aggregates. (The biggest difference license-compatibility wise is the Apache license compatibility.)

catch’s picture

I think we should explicitly add licenses for external libraries even if it's GPL2, otherwise you're left wondering.

wim leers’s picture

#13: indeed.

So, any other concerns? (I think I addressed all feedback so far?)

Bojhan’s picture

Its unclear to me if there are UI changes. None are visible, but we do add a page?

wim leers’s picture

#15: Correct. At /system/jslicense.

catch’s picture

The state entry can't necessarily be relied upon - at least the proposal to aggregate based on libraries in #1014086: Stampedes and cold cache performance issues with css/js aggregation would remove the need to write aggregates to state since we could determine the aggregate to serve based entirely on URL. Even if we don't fix that in core before 8.0.0, the system is pluggable (i.e. agrcache and advagg both swap it out for 7.x already and neither use the core aggregate variable).

@Bojhan yes this will add a page, and possibly a visible link to that page on every other.

wim leers’s picture

#17:

If #1014086: Stampedes and cold cache performance issues with css/js aggregation lands, we could just list all JS assets in the files/js directory; they'd all have the GPL2+ license anyway. We could then render cache for, say, 5 minutes (or more), to prevent FS hits every time /system/jslicense is loaded.

However, you're absolutely right that asset aggregation plugin overrides would break this. I don't see any other way of fixing that other than adding a new API to list all known aggregates, unfortunately.
Which makes me think we should we descope this issue a little bit: introduce license metadata for asset libraries here, expose /system/jslicense for each JS asset, but don't provide license information for aggregates just yet, because that requires a new API.

Thoughts?

wim leers’s picture

Marked #156124: JS and CSS aggregation deletes license information as a duplicate.

Can I has reviews? :)

hass’s picture

It sounds really bad to me that incompatible licenses should not be aggregated together in one file and causing several http requests in a browser. That may makes me setting a non-gpl libs as true just to make it compressed in one file. The intention of #156124: JS and CSS aggregation deletes license information was to keep the license above the code (remote value). Performance is more important.

wim leers’s picture

Title: Implement JS Web License Labels to allow for JS minification; each asset library should declare its license » Implement JS Web License Labels to allow for JS minification
Issue summary: View changes
Status: Needs review » Postponed
Issue tags: -beta target, -sprint
StatusFileSize
new37 KB

#20: You're right.

So, this patch essentially still has two problems:

  1. reliance on the system.js_optimized_aggregate_files entry in State, which shouldn't be relied upon because it's an overridable implementation detail (#17)
  2. negative performance impact when a site uses non-GPL-compatible JS (#20)

I propose we split this issue up in at least two parts:

  1. Introduce the license metadata for *.libraries.yml files — i.e.: the API change.
  2. Implement JS Web License labels, for which we will still need to solve problems 1 and 2 above — this will not change APIs.

That way, we increase our chances of success. I think all of us want to see Drupal 8 ship with JS minification. So let's get the first point done.


I've opened #2276219: Asset libraries should declare their license and posted a patch. Please review that!

This issue is now blocked on #2276219. Attached is a straight reroll of #12, which should not be reviewed — I'm only posting to make it easier to continue working on this once #2276219 lands.

hass’s picture

We also do not care about CSS framework licenses for many years and just remove all comments (including the license comments). This cannot be a blocker for JS minification. Otherwise we need to remove CSS aggregation also as long as this feature is in. The ONLY reason why we have removed JS minification from core in past was that *valid* JS code has been destroyed by the minification.

I can search for the case ID if this is of interest.

wim leers’s picture

Title: Implement JS Web License Labels to allow for JS minification » [PP-1] Implement JS Web License Labels to allow for JS minification
Related issues: +#2307419: AssetCollectionOptimizerInterface should allow listing and deleting all aggregates (optimized collection assets)
StatusFileSize
new15.22 KB

#2276219: Asset libraries should declare their license landed! This issue is now unblocked :)

Attached is a straight reroll of #21, but adjusted for #2276219: Asset libraries should declare their license and other changes in HEAD.


#22: there are a few important differences:

  1. Even minified CSS can quite easily be "unminified", the only thing missing then are comments. By its very nature, it cannot be obfuscated (e.g. variable names being renamed to be nonsensical), because CSS selectors and properties need to be interpreted by the browser.
  2. CSS does not contain logic/executable code (except in rare cases "logic" in the form of the calc() property, but that's always trivial). I think that the FSF therefore considers the inability to obfuscate CSS (i.e. you can always access the source code) in combination with the Linking Exception, this continues to meet the GPL's requirements (provided that you also meet, for each linked independent module, the terms and conditions of the license of that module).
  3. For more details, see https://www.gnu.org/philosophy/javascript-trap.html

Also see #1649654: Non-trivial JavaScript files need GPL license declaration for compliant distribution to browsers, #1536810: Adhere to licensing issues when minifying files and #1649670: [META] Improving Drupal sites JavaScript Licence Compliance — as well as other issues where this has been discussed. This has been a known problem for many years, and was discussed by many, it's not just a claim by me!


To address #17, i.e. to fix the reliance on the system.js_optimized_aggregate_files entry in State, which shouldn't be relied upon because it's an overridable implementation detail problem, I propose we expand AssetCollectionOptimizerInterface slightly, to allow for listing and deleting all aggregates. See #2307419: AssetCollectionOptimizerInterface should allow listing and deleting all aggregates (optimized collection assets) for that.

So once again, this issue is blocked, but this should be the last time.

hass’s picture

There is one thing I cannot find in the patch. How can I define that the file is already a minified version? In case a file is already minified it is a waste of cpu to double minify them. I think we need to combine them in such a case only.

wim leers’s picture

Great point. I hadn't considered that.

The reason I hadn't considered it yet, is precisely because of what you say: we don't have a flag to indicate that a file is already minified.
Personally, I think that all JS files we ship with should be unminified. Minification should be done by Drupal's JS minifier (which could use UglifyJS or whatever external library is best or carries your preference). Because only then you can easily switch to the unminified version, and hence only then you can easily debug.

In any case, what you're calling out is something that's out of scope for this issue. As it stands, Drupal 8 contrib modules that add a JS minifier would have to GUESS based on heuristics (file name, first 100 bytes, …) whether a JS file is already minified or not. The only reason it's not a problem in HEAD is because we still don't have a JS minifier!
I opened an issue to discuss that: #2307717: Inform Drupal in *.libraries.yml via a new per-JS file "minified" flag whether a JS file has been minified.

hass’s picture

I'm sorry to say, but you are wrong. Check out the assets folder. Ckeditor, html5shiv and others are minified. I also think it's wrong to use UglifyJS in all cases as the maintainers of libs have made extreme tests with several minifiers and decided for the best from their test results, latency, ux and other reasons. If you get this in here without such a "disable" directive we will double minify a lot of code.

I have not seen these minifier code, but minifying 5mb js code on the fly sounds like a crazy task to me. Isn't UglifyJS not a nix only tool, too? This is a showstopper to me.

wim leers’s picture

Title: [PP-1] Implement JS Web License Labels to allow for JS minification » Implement JS Web License Labels to allow for JS minification
Issue summary: View changes
Status: Postponed » Needs review
StatusFileSize
new16.89 KB
new15.03 KB

Hurray, #2307419: AssetCollectionOptimizerInterface should allow listing and deleting all aggregates (optimized collection assets) has landed; which was the last blocker (see #23)! Rerolled the patch.


Already since the reroll in #23, the "only allow gpl-compatible JS files in the aggregate" part of the original patch has been removed. I've now also updated the issue summary to reflect that.


As of #25/#26, we have a new soft blocker, which is: Drupal core doesn't yet allow us to know which files already are minified.

(Yes, hass, you were absolutely right that I'm wrong. CKEditor and others are minified. Of those, I think only CKEditor is rightfully minified, because minifying that is a massive amount of work (because it's a massive amount of JS) and you rarely need to debug it (only when developing CKEditor plugins). html5shiv could also be acceptable because it's extremely, extremely rare that one would need to debug that, plus one doesn't develop against that. But one needs to step through jQuery's code quite frequently, so hence it shouldn't be minified.
I also gave a bad example — Drupal of course wouldn't ship with UglifyJS, because that'd imply a dependency on node.js.
All that being said: without actually having a JS minifier in core, my "should not be minified" reasoning is pointless.)

I think it's a soft blocker, because the double minification really is an existing bug in Drupal core; it already happens in HEAD!

Therefore I think we can continue with this patch just fine now. To fix the double minification problem, I rolled another patch: #2307717-13: Inform Drupal in *.libraries.yml via a new per-JS file "minified" flag whether a JS file has been minified.

nod_’s picture

I'm happy with the patch. Only got a few comments:

Having a (hidden) link at the bottom of the page is not great, especially because I'm guessing people will end up blacklisting /system/jslicense in their robots.txt (because it gives away most of the modules used on the site (security issue too maybe?)).

Having a <link rel="jslicense" href=""> would serve the same purpose. I'm reaching out to the relevant mailing list to know what they'll say.

Other than that the PHP looks good to me, not that I'm a great judge for that :D

nod_’s picture

Status: Needs review » Needs work

Looks like the link stuff instead of the a tag is supported by the librejs firefox extension http://lists.gnu.org/archive/html/help-librejs/2014-02/msg00004.html (and at least unofficially supported). Couldn't get a definite yes/no for now but I say let's just go with <link>

wim leers’s picture

Yes, let's do that :) I always disliked the "hidden link at the bottom of the page" approach. Thanks for the feedback!

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new16.67 KB
new1.43 KB

Implemented #28/#29.

wim leers’s picture

Issue summary: View changes
moshe weitzman’s picture

Status: Needs review » Reviewed & tested by the community

Our way to add [link] tag is ugly but thats not invented here. All feedback has been addressed. PHP code looks good.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs reroll

Needs a reroll

git ac https://www.drupal.org/files/issues/js_license_web_labels-2258313-31.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17065  100 17065    0     0   104k      0 --:--:-- --:--:-- --:--:--  130k
error: patch failed: core/modules/system/src/Tests/Common/JavaScriptTest.php:32
error: core/modules/system/src/Tests/Common/JavaScriptTest.php: patch does not apply

The last submitted patch, 31: js_license_web_labels-2258313-31.patch, failed testing.

wim leers’s picture

Status: Needs work » Reviewed & tested by the community
Issue tags: -Needs reroll
StatusFileSize
new16.67 KB

Straight reroll, with thanks to git's 3-way merge.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. <tr id="core-assets-vendor-ckeditor-ckeditorjs" class="odd">
      <td><a href="http://drupal8alt.dev/core/assets/vendor/ckeditor/ckeditor.js"><code>ckeditor.js</code></a></td>
      <td><a href="https://github.com/ckeditor/ckeditor-dev/blob/887d81ac1824008b690e439a1b29eb4f13b51212/LICENSE.md">GNU-GPL-2.0-or-later</a></td>
      <td class="priority-medium"><a href="http://drupal8alt.dev/core/assets/vendor/ckeditor/ckeditor.js"><code>ckeditor.js</code></a></td>
    </tr>
    

    I'm not sure this row is correct - if I understand https://www.gnu.org/licenses/javascript-labels.html correctly the third row should be a link to a non minified - source code version of the javascript. Maybe the libraries info needs to have a link we can use here.

  2. +++ b/core/lib/Drupal/Core/Asset/JsLicenseWebLabelsAnnotator.php
    @@ -0,0 +1,53 @@
    +      throw new \Exception('Only file JavaScript assets can be optimized.');
    ...
    +      throw new \Exception('Only file JavaScript assets with preprocessing enabled can be optimized.');
    

    Exceptions should not have fullstops and should we be using the general exception class here?

  3. +++ b/core/lib/Drupal/Core/Asset/JsLicenseWebLabelsAnnotator.php
    @@ -0,0 +1,53 @@
    +    // Generate a prefix to be prepended to the "optimized" asset (no actual
    +    // optimizations are made; this is a no-op optimizer) that deep-links to the
    +    // license information on the JavaScript License Web Labels page.
    +    $url = $this->urlGenerator->generateFromRoute('system.javascript_license_web_labels', array(), array('fragment' => drupal_clean_css_identifier($js_asset['data'])));
    +    $prefix = "/** JavaScript asset: " . $js_asset['data'] . '; for license information, see ' . $url  . "  **/\n\n";
    +    return $prefix . file_get_contents($js_asset['data']);
    

    We seem to be lacking test coverage of this.

  4. +++ b/core/modules/system/src/Controller/AssetLicenseInfoController.php
    @@ -0,0 +1,182 @@
    +use Drupal\Component\Utility\String;
    

    Not used

  5. +++ b/core/modules/system/system.routing.yml
    @@ -418,3 +418,11 @@ system.admin_content:
    +system.javascript_license_web_labels:
    +  path: '/system/jslicense'
    

    We have no test that tests this route - afaics.

edit: fix html

wim leers’s picture

Status: Needs work » Postponed

#37.1: You're right. That was another reason I originally had "no minified JS at all in core" as one option to fix #2307717: Inform Drupal in *.libraries.yml via a new per-JS file "minified" flag whether a JS file has been minified. But since I was out of this issue and that issue for some time, I must've forgot about that. catch made it clear at https://www.drupal.org/node/2307717#comment-8990911 that we want minified JS in core. So that makes me think the only way to implement JS License Web Labels, is by linking to a CKEditor-hosted version of the source, i.e. by requiring a unminified-source property that links to the unminified source for JS assets that are minified (that have the minified flag set to true).
That means #2307717: Inform Drupal in *.libraries.yml via a new per-JS file "minified" flag whether a JS file has been minified will have to land before I can finish this.

wim leers’s picture

lauriii’s picture

Status: Needs work » Needs review
StatusFileSize
new15.5 KB

Rerolled last patch.

Status: Needs review » Needs work

The last submitted patch, 40: js_license_web_labels-2258313-40.patch, failed testing.

tstoeckler’s picture

  1. +++ b/core/lib/Drupal/Core/Asset/JsLicenseWebLabelsAnnotator.php
    @@ -0,0 +1,53 @@
    +      throw new \Exception('Only file JavaScript assets can be optimized.');
    ...
    +      throw new \Exception('Only file JavaScript assets with preprocessing enabled can be optimized.');
    

    Would be slightly more semantic to throw an \InvalidArgumentException here

  2. +++ b/core/modules/system/system.module
    @@ -626,6 +626,12 @@ function system_page_attachments(array &$page) {
    +  // Add the JavaScript Web License Labels link to all pages.
    

    If we don't send any JavaScript at all (i.e. for anonymous users), this shouldn't be added either, right?

yesct’s picture

Issue tags: -front-end performance +frontend performance

changing to use the more common tag, so the less common one can be deleted, so it does not show up in the auto complete and confuse people.

rainbowarray’s picture

I just want to confirm a couple of items:

1) Is it now resolved so that all JS can be aggregated in one file regardless of license type?

2) Have we verified that having the link element to the JS license won't result in a file download?

Those seem like two key things for front-end performance with this issue.

wim leers’s picture

Version: 8.0.x-dev » 8.1.x-dev

We should be focused on shipping D8 right now, and only working on non-criticals that have long-term consequences for D8. This issue can easily be done in 8.1; the necessary API changes have already been made.

Hence updating the "version" component of this issue.

webchick’s picture

While that is true, I think this is still worth working on in 8.0.x, since:

a) It's an enabler for a performance improvement, and we need more of those. ;)
b) (raised today by nod/jhodgdon) If we try and do JSDoc, without this we're going to bloat the size of JS files unnecessarily.

webchick’s picture

Version: 8.1.x-dev » 8.0.x-dev

Doing that. :)

nod_’s picture

Assigned: wim leers » Unassigned

While it'd be cool, let's not expect wimleers to fix all our bugs.

wim leers’s picture

:P

lauriii’s picture

Maybe we can clone Wim Leers and then he could fix all the bugs?

lauriii’s picture

Status: Needs work » Needs review
StatusFileSize
new16.01 KB

Rerolling again

Status: Needs review » Needs work

The last submitted patch, 51: implement_js_web-2258313-51.patch, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

damienmckenna’s picture

Status: Needs work » Needs review
StatusFileSize
new1.18 KB
new4.86 KB
new14.48 KB

WIP reroll.

The changes to template_preprocess_html() didn't apply as it has changed substantially since 8.0.0-beta11. I've attached one file of the rejected changes to the theme.inc file and another with beta11's template_preprocess_html with the patch applied, so it can be seen in context.

Status: Needs review » Needs work

The last submitted patch, 55: drupal-n2258313-55.patch, failed testing.

wim leers’s picture

Thanks for picking this up again!

damienmckenna’s picture

Might someone who knows the JS aggregation code be able to take a look at re-applying the missing template_preprocess_html() change? Thanks.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev
mfb’s picture

Issue tags: -JavaScript +JavaScript

By the way, I did a very minimal port of https://www.drupal.org/project/librejs from Drupal 7 to Drupal 9. So far, it's missing much of the functionality of the Drupal 7 version, like allowing administrators to configure the license and source code URL of JS resources. However, at least it does grab the license and source metadata of each JS resource automatically from core library services, which is a nice starting point, and if anyone's interested they can add/restore more functionality... :)

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

catch’s picture

Coming here from #1014086: Stampedes and cold cache performance issues with css/js aggregation and #3301573: Remove the aggregate stale file threshold and state entry.

I'd like to remove the state entries for js and css assets, since it's no longer necessary for the implementation at all. However this means that the implementation in this issue would be broken.

However, I can't see why the complexity added here is necessary at all.

From https://www.gnu.org/licenses/javascript-labels.html:

Add a page for JavaScript license web labels to your site. You can use whatever path or filename is most convenient for you; others will find it through links. The page must include one table marked with the attribute id="jslicense-labels1". This name lets automated tools find the table easily, and tells them what format to expect. Each row of this table will contain three cells, providing information about a standalone JavaScript file used on the site, its license, and how visitors can obtain its source code.

This isn't saying you need to provide a list of every aggregate on the website, it's saying you need to provide a list of every standalone file on the site + a link to its source code.

So I think what is necessary is actually this:

1. Add license + source code link to every asset in a library definition.
2. On the weblicense page, list the standalone js file in core (i.e. what the library definition includes), the license, and a link to the source code.

For files that we include pre-minified in core (like ckeditor), we'd need to link to an unminified version of the file (either one we include or to the repo).

The fact that we'll be bundling files together and minifying them is irrelevant to the fact that this page will list every file that's used on the site - which it still will because we'll be getting it from the library definitions.

Should also make the page itself a lot easier to read for humans.

mfb’s picture

The LibreJS browser extension did require aggregated/bundled JS to be listed, as it would only load JS files that were listed (as free software) on the jslicense page. However, I have no idea if anyone uses that extension nowadays.

catch’s picture

The LibreJS browser extension did require aggregated/bundled JS to be listed, as it would only load JS files that were listed (as free software) on the jslicense page.

Since core doesn't implement a jslicense page, everything would already be broken for someone using that extension, even if serving unminified, unbundled core JavaScript. I think we need to meet the licensing requirements but not a particular technical implementation that's not friendly to humans.

ravi.shankar’s picture

StatusFileSize
new14.37 KB
new6.27 KB

Added reroll of patch #55 on Drupal 9.4.x.

nod_’s picture

It seems like there are alternative methods to declare the licence, where a dedicated page is not necessary, see The "other" section of https://www.gnu.org/software/librejs/free-your-javascript.html#magnet-li...

as stated in the JSWL page the table is a good solution in the following situation (emphasis mine):

If you are a webmaster deploying minified JavaScript on a site, here's a method for stating their licenses and source code locations without altering the minified (or aggregated in our case) files themselves. It's especially helpful in cases where the JavaScript is under one of the GNU licenses, but does not include the additional permission proposed in Section 3.2 of Setting Your JavaScript Free, by Loic Duros.

I don't have a problem with messing with the aggregate contents and wrapping an already minified code with a license comment such as

// @license [magnet-link] [human readable name of the license]
... JS code
// @license-end

That avoid many problems with declaring a new route, hidden link, new meta etc. it's directly in the file. That would make aggregated JS more compliant that non aggregated JS too (since libs like shepherd don't put their license in the minified version).

Also librejs is a tool that prevent the execution of "non-free" javascript, and they explicitly state that it's not for security purposes (see disclaimer in https://www.gnu.org/software/librejs/manual/librejs.html), or even checking that the licences are properly declared. It's simply to not run code that is not "free". Anyone is free to use it but I don't think we should bend over backwards to ensure that it's working with that extension.

catch’s picture

// @license [magnet-link] [human readable name of the license]
... JS code
// @license-end

This looks like a good option to me, easy to wrap the file contents in the optimizer.

#2276219: Asset libraries should declare their license added the license information to the library rather than the file, but aggregation deals with files (derived from libraries), usually this is fine but it's not guaranteed to be. If a library was combining assets with two different licenses, maybe that should just be two libraries and we don't worry about it? Do we need to go back through the library definitions and add magnet links?

nod_’s picture

I wouldn't worry about having files with different licences in the same library, if that's the case the rest of the metadata is probably wrong too, such as the version

ravi.shankar’s picture

StatusFileSize
new14.07 KB
new9.08 KB

Fixed Drupal CS issues of patch #72.

nod_’s picture

@ravi we're going to change the direction of the patch, there is no need to reroll the previous one as we won't be using that approach.

mfb’s picture

The magnet-link part of the spec seems kinda wonky (ah, gnu). To streamline everything for library authors, they could be automatically added by core based on the license identifier?

nod_’s picture

Status: Needs work » Needs review
StatusFileSize
new4.07 KB

The original solution can't work since we aggregate files with different licenses. like jquery is MIT and our code is GPLv2 and they're in the same file, so we need to use inline license declaration, which makes things easier for us so yay.

I ignored the whole magnet part like you said it's kinda wonky and I don't see the real value, we have the URL to the actual licenses if necessary. I checked the librejs code to see how it'd work and they hardcode the possible licenses and associated "valid" URLs here https://git.savannah.gnu.org/cgit/librejs.git/tree/license_definitions.j... So the SPDX identifier is not enough for this extention, but it is enough for humans

Went with /* */ type comments so that JS and CSS license comments end up the same.

nod_’s picture

StatusFileSize
new3.93 KB

A version adding the license URL to the license comment.

The last submitted patch, 79: core-jswl-2258313-79.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 80: core-jswl-2258313-80.patch, failed testing. View results

mfb’s picture

In addition, it sounds like LibreJS cannot handle multiple @license tags per script, as they say "Please note that no code or comment should be added to the script after the // @license-end comment. However, whitespace is allowed."

nod_’s picture

StatusFileSize
new3.14 KB

oh right, well, let's simplify all this a bit then. Removed the @license-end part

to come back to #74

If a library was combining assets with two different licenses, maybe that should just be two libraries and we don't worry about it?

Turns out this is already the case in core, as we include jQuery UI files in a Drupal library (dialog/autocomplete) so we have files with GPLv2 and MIT in the same library definition. jQuery UI keeps some license information in all their files so things are still accurate. Since we already covered the fact that existing automated tools won't be able to understand our aggregate I don't think we should care much.

catch’s picture

Dialog and autocomplete are an anomaly in 10.x since we explicitly munged everything together to deprecate the jQuery UI libraries. Given we'd normally have separate definitions seems fine to make that assumption. Especially since jquery ui has the license info anyway.

The inline version of this seems really straightforward and a good way to do it.

nod_’s picture

Status: Needs work » Needs review
StatusFileSize
new10.9 KB
new7.37 KB

fixing tests

catch’s picture

Version: 9.4.x-dev » 10.1.x-dev

The last submitted patch, 86: core-jswl-2258313-86.patch, failed testing. View results

nod_’s picture

StatusFileSize
new14.51 KB
new3.19 KB

need to update deprecated code path

wim leers’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Wow, exciting to see this move forward again, and loving how much simpler this is!

Not only simpler, but also more performant, because now we're optimizing for front-end performance instead of optimizing for the simplest possible LibreJS implementation.

  1. +++ b/core/lib/Drupal/Core/Asset/AssetResolver.php
    @@ -138,6 +138,8 @@ public function getCssAssets(AttachedAssetsInterface $assets, $optimize, Languag
    +          // Copy the Drupal library license information on each file.
    
    @@ -244,6 +246,8 @@ public function getJsAssets(AttachedAssetsInterface $assets, $optimize, Language
    +            // Copy the Drupal library license information on each file.
    

    🤓 Nits:

    s/Drupal library/asset library/

    s/on/to/

  2. +++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizer.php
    @@ -124,7 +124,13 @@ public function optimize(array $css_assets, array $libraries) {
    +                // Ensure license information is kept after optimization.
    
    +++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
    @@ -160,7 +160,13 @@ public function deleteAll() {
    +      // Ensure license information is kept after optimization.
    
    +++ b/core/lib/Drupal/Core/Asset/JsCollectionOptimizer.php
    @@ -124,7 +124,13 @@ public function optimize(array $js_assets, array $libraries) {
    +                // Ensure license information is kept after optimization.
    
    +++ b/core/lib/Drupal/Core/Asset/JsCollectionOptimizerLazy.php
    @@ -172,7 +172,13 @@ public function deleteAll() {
    +      // Ensure license information is kept after optimization.
    

    🤔 I think instead of "kept" it'd be clearer if we'd write "available as a comment"?

  3. +++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizer.php
    @@ -124,7 +124,13 @@ public function optimize(array $css_assets, array $libraries) {
    +                  $data .= "/* @license " . $css_asset['license']['name'] . " " . $css_asset['license']['url'] . " */\n";
    

    🤔 I think it'd be a lot clearer (to humans especially) if we'd prefix this with \n?

  4. The sole test coverage that proves that this works uses css_input_with_import.css.optimized.aggregated.css for the expectation, but it does not test the case of multiple licenses within a single aggregate. I think we want explicit test coverage for that.

After that test coverage is added, I think this is pretty much ready to go 😮

nod_’s picture

Status: Needs work » Needs review
StatusFileSize
new29.27 KB
new22.82 KB

1. fixed
2. fixed
3. Went a different way, the problem is when processing @import rules in the CSS optimizer. Adding a leading "\n" would start all js and css with a blank line. probably safe but you never know, i personally don't like starting with a blank line in those files.
4. added a new method to test the licenses specifically.

nod_’s picture

Title: Implement JS Web License Labels to allow for JS minification » Add license information to aggregated assets

we're not actually implementing js web labels as defined (we don't have the magnet links) so renaming the issue.

mfb’s picture

What do you think about also adding the "remote" URL, as it provides access to the source code for minified JS?

nod_’s picture

I think it's out of scope, we link to the project license, which is a link to the source code itself, so we're kinda giving the repo link implicitly.

also remote is outside of the "license" key in the libraries, so that would mean adding additional informations to the individual asset array.

wim leers’s picture

Status: Needs review » Needs work
Issue tags: -Needs tests
  1. +++ b/core/tests/Drupal/Tests/Core/Asset/CssCollectionOptimizerLazyUnitTest.php
    @@ -72,4 +79,74 @@ public function testCssImport(): void {
    +    $license = [
    
    +++ b/core/tests/Drupal/Tests/Core/Asset/CssCollectionOptimizerUnitTest.php
    @@ -64,20 +64,96 @@ public function testCssImport() {
    +    $license = [
    ...
    +    $license = [
    

    Übernit: s/$license/$gpl_license/

  2. +++ b/core/tests/Drupal/Tests/Core/Asset/css_test_files/css_license.css.optimized.aggregated.css
    @@ -1,4 +1,6 @@
    +/* @license GNU-GPL-2.0-or-later https://www.drupal.org/licensing/faq */
    
    @@ -12,3 +14,8 @@ body{margin:0;padding:0;background:#edf5fa;font:76%/170% Verdana,sans-serif;colo
    +/* @license MIT https://opensource.org/licenses/MIT */
    

    👍 This is the test coverage we needed!

ravi.shankar’s picture

Status: Needs work » Needs review
StatusFileSize
new29.32 KB
new5.72 KB

Addressed comment #95.1

catch’s picture

  1. +++ b/core/lib/Drupal/Core/Asset/JsCollectionOptimizer.php
    @@ -124,7 +124,14 @@ public function optimize(array $js_assets, array $libraries) {
                   foreach ($js_group['items'] as $js_asset) {
    +                // Ensure license information is available as a comment after
    +                // optimization.
    +                if ($js_asset['license'] !== $current_license) {
    +                  $data .= "/* @license " . $js_asset['license']['name'] . " " . $js_asset['license']['url'] . " */\n";
    +                }
    +                $current_license = $js_asset['license'];
                     // Optimize this JS file, but only if it's not yet minified.
                     if (isset($js_asset['minified']) && $js_asset['minified']) {
    

    Shouldn't this only be added if the asset is unminified? Otherwise if the minified file has it anyway, we might be adding it twice?

    But then if minified files don't include it, it's adding missing information.

    So.. no idea tbh but better ask.

  2. +++ b/core/lib/Drupal/Core/Asset/JsCollectionOptimizerLazy.php
    @@ -172,7 +172,14 @@ public function deleteAll() {
         $data = '';
    +    $current_license = FALSE;
         foreach ($group['items'] as $js_asset) {
    +      // Ensure license information is available as a comment after
    +      // optimization.
    +      if ($js_asset['license'] !== $current_license) {
    +        $data .= "/* @license " . $js_asset['license']['name'] . " " . $js_asset['license']['url'] . " */\n";
    +      }
    +      $current_license = $js_asset['license'];
           // Optimize this JS file, but only if it's not yet minified.
           if (isset($js_asset['minified']) && $js_asset['minified']) {
             $data .= file_get_contents($js_asset['data']);
    

    Same question here.

longwave’s picture

+++ b/core/tests/Drupal/Tests/Core/Asset/css_test_files/css_license.css.optimized.aggregated.css
@@ -12,3 +14,8 @@ body{margin:0;padding:0;background:#edf5fa;font:76%/170% Verdana,sans-serif;colo
+body{margin:0;padding:0;background:#edf5fa;font:76%/170% Verdana,sans-serif;color:#494949;}.this .is .a .test{font:1em/100% Verdana,sans-serif;color:#494949;}.this
+.is
+.a
+.test{font:1em/100% Verdana,sans-serif;color:#494949;}some :pseudo .thing{border-radius:3px;}::-moz-selection{background:#000;color:#fff;}::selection{background:#000;color:#fff;}@media print{*{background:#000 !important;color:#fff !important;}@page{margin:0.5cm;}}@media screen and (max-device-width:480px){background:#000;color:#fff;}textarea,select{font:1em/160% Verdana,sans-serif;color:#494949;}

This looks like duplicate CSS added to the file? Is this a bug? I don't see where this is otherwise added to a test case.

nod_’s picture

#97 Minified version don't all include license, see once, jquery, loadjs, or shepherd minified files. CKE5 include copyright info, not license info in their minified files. So it doesn't hurt to always add it to be sure. Also our CSS minifier removes all comments, including license info that we have in normalize.css so we need to add it back or change the optimizer. This ensure a standard way of declaring the license too instead of relying on the third party.

#98 it's a test file that only serve to ensure the aggregation works as expected. It doesn't matter what the content are, that threw me off a bit at first too. The import part of this file is the 2 different licenses being shown in the aggregate. for simplicity I used the same file and declared a different license because I already had the minified version of that css file on hand.

longwave’s picture

Re #98 I see now that it's a brand new test case and that file is just reusing parts of an older test case, the explanation makes sense.

nod_ credited Owen Barton.

nod_’s picture

still need a RTBC :D

porting issue credit from a bunch of duplicate issues.

longwave’s picture

Status: Needs review » Reviewed & tested by the community

Sorry, should have done that in #100 :)

  • catch committed 53a59af on 10.1.x
    Issue #2258313 by Wim Leers, nod_, ravi.shankar, lauriii, catch, mfb,...
catch’s picture

Status: Reviewed & tested by the community » Fixed
+++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
@@ -160,7 +160,14 @@ public function deleteAll() {
     $data = '';
+    $current_license = FALSE;
     foreach ($group['items'] as $css_asset) {
+      // Ensure license information is available as a comment after
+      // optimization.
+      if ($css_asset['license'] !== $current_license) {
+        $data .= "/* @license " . $css_asset['license']['name'] . " " . $css_asset['license']['url'] . " */\n";
+      }
+      $current_license = $css_asset['license'];
       $data .= $this->optimizer->optimize($css_asset);
     }

Only question I had here was whether we should be doing this inside CssOptimizer instead of CssOptimizerLazy, for example in ::processFile(). However, for JavaScript preprocessing and minification are separate (i.e. we only run JsOptimizer on non-minified files), so... answered my question and the collection optimizers are the right place.

Committed 53a59af and pushed to 10.1.x. Thanks!

This is theoretically backportable to 9.5.x/10.0.x, but it'd need a re-roll due to aggregation class changes, so I think it's easier to leave in 10.1.x since the issues it blocks will only land in 10.1.x too.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

jweowu’s picture

It seems to me that this issue/fix does not address the problem of providing end-users with a reference to the source code for the aggregated GPL'd asset files.

Can someone please confirm that this is still an outstanding issue?

As such code is "distributed" by necessity (to the end-user's machine, where it is executed), GPL'd javascript must also provide the user with access to the source code, and minified and/or aggregated derivatives are not sufficient for that requirement (the GPL defines source code as the preferred form of the work for making changes in it).

Section 3.2.4 "Stylized comment" near the bottom of https://www.gnu.org/software/librejs/free-your-javascript.html recommends the use of a @source indicator in the header comment at the top of a minified JS file:

/**
 *
 * @source: http://www.lduros.net/some-javascript-source.js
 *
 * @licstart  The following is the entire license notice for the 
 *  JavaScript code in this page.
 *
 * [...]
 *
 * @licend  The above is the entire license notice
 * for the JavaScript code in this page.
 *
 */

We're using @license which sounds fine for indicating that bit; but I can't see anything in the committed change (or the files I've checked in https://git.drupalcode.org/project/drupal/-/tree/10.1.x/core/lib/Drupal/... ) for adding @source comments.

I believe a header comment for all aggregated or minified files should display a @source value for each GPL'd asset, providing access to the original source file.

I couldn't find a more relevant issue than this one, so we likely need a new issue to be raised; but this seems like the right place to ask for initial comments.

mfb’s picture

@jweowu see #94 - adding an additional source link was considered out-of-scope of this issue, but you could open a new issue.

Many libraries are built from a whole project, not just one "source file" , so I'd say the "remote" URL that is already available in the *.libraries.yml files could be used as @source - it's basically a link to "the preferred form of the work for making modifications to it" (quoting GPL).

jweowu’s picture