Problem/Motivation

This is a blocker for #2368797: Optimize ajaxPageState to keep Drupal 8 sites fast on high-latency networks, prevent CSS/JS aggregation from taking down sites and use HTTP GET for AJAX requests.

On top of that, we should do this for consistency, just like we've done #2377397: Themes should use libraries, not individual stylesheets.

This is blocked on #2378095: Convert all remaining attached individual CSS/JS assets to attached asset libraries and #2382557: Change JS settings into a separate asset type.

Proposed resolution

Only allow assets to be attached via the asset library system.

Remaining tasks

Build consensus & review.

User interface changes

None.

API changes

  • No longer able to use the (already deprecated) attaching of individual assets via $build['#attached']['css'] and $build['#attached']['js']
  • Remove support for inline CSS and JS assets (see #5.3 + #8 for details).

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue priority Critical because it blocks the critical perf issue #2368797: Optimize ajaxPageState to keep Drupal 8 sites fast on high-latency networks, prevent CSS/JS aggregation from taking down sites and use HTTP GET for AJAX requests.
Prioritized changes The main goal of this issue is performance/removing previously deprecated code.
Disruption Very low disruption: it primarily enforces prior policy changes ("assets should be attached via asset libraries"). Secondarily, it also removes support for inline CSS/JS assets, which have always been discouraged, and are easy to convert into file assets.

Comments

wim leers’s picture

Title: [PP-1] Attach assets only via the asset library system » [PP-2] Attach assets only via the asset library system
Issue summary: View changes
wim leers’s picture

wim leers’s picture

Title: [PP-2] Attach assets only via the asset library system » [PP-1] Attach assets only via the asset library system
wim leers’s picture

Assigned: Unassigned » wim leers
Status: Postponed » Active

With #2382557: Change JS settings into a separate asset type being very close to RTBC, I've already begun working on this patch.

wim leers’s picture

About this patch, besides the obvious changes:

  1. The majority of this patch is for AttachedAssetsTest, which is a hugely cleaned up version of CascadingStylesheetsTest and JavaScriptTest (which still contained things like #attached_library, tests that \Drupal\system\Tests\Asset\CssGrouperUnitTest already did… but more comprehensively and as a unit test
  2. Another big part: half of MergeAttachmentsTest was deleted.
  3. And the third big part: this patch explicitly removes support for inline CSS and JS. It's possible to support it in libraries, but after discussing it with the principal JS maintainer (nod_), it turns out the plan was all along to not support this anymore. There are no good use cases for either to be first-class citizens. As documented at https://www.drupal.org/node/2216195, if you have inline JS, just put it in a block or directly in a Twig template. (Interdiff for only this part included to make it easy to exclude it from the patch.)

The smaller patch is the actual one, but for now it needs to include #2382557: Change JS settings into a separate asset type.

Status: Needs review » Needs work

The last submitted patch, 5: assets_only_libraries-2382533-5_on_top_of_2382557.patch, failed testing.

wim leers’s picture

How this manages to cause 2 failures (1 locally) in \Drupal\comment\Tests\CommentTokenReplaceTest is beyond me.

I'll just wait for #2382557 to land before debugging this further.

catch’s picture

Agreed on removing inline js. When I've done site audits a very common problem is a poorly written contrib or custom module sticking inline js on the page instead of using Drupal.settings. Then that inline js because it is added in amongst all the other js on the page results in two aggregates (one before the inline js, one after) where there originally would have been one. So supporting that as a regular asset type makes it very easy to mess up the front end performance of a site.

I wonder a little bit about inline CSS - some rendering strategies try to group layout CSS inline so that it gets downloaded very, very quickly as part of the initial HTML request. I guess that's something that would happen in the css aggregation/rendering stages anyway though and no way a module can do this in #attached anyway.

wim leers’s picture

I wonder a little bit about inline CSS - some rendering strategies try to group layout CSS inline so that it gets downloaded very, very quickly as part of the initial HTML request. I guess that's something that would happen in the css aggregation/rendering stages anyway though and no way a module can do this in #attached anyway.

Exactly!
It'd have to be an alternative implementation of the CSS asset renderer service (AssetCollectionRendererInterface), which is totally possible in D8 :)

mikeytown2’s picture

I agree that inline css/js messes up aggregates, but this can be fixed; in advagg there is a setting that forces all inline code to be at the bottom, there is another setting that forces all external code to be at the top, and a 3rd setting that will group all browser conditionals together; these are all things that can mess up aggregates. Having a standard way to add in inline code is nice as it allows one an easy way to modify it; be it minification or delayed execution or modification of the code if it was added by a module.

wim leers’s picture

Title: [PP-1] Attach assets only via the asset library system » Attach assets only via the asset library system

#2382557: Change JS settings into a separate asset type was committed! Now this is unblocked.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new111.85 KB

Straight reroll of assets_only_libraries-2382533-5-do-not-test.patch in #5.

berdir’s picture

+++ b/core/modules/history/history.module
@@ -135,14 +135,11 @@ function history_cron() {
   if (($display->originalMode === 'full') && \Drupal::currentUser()->isAuthenticated()) {
-    // When the window's "load" event is triggered, mark the node as read.
-    // This still allows for Drupal behaviors (which are triggered on the
-    // "DOMContentReady" event) to add "new" and "updated" indicators.
-    $build['#attached']['js'][] = array(
-      'data' => 'window.addEventListener("load",function(){Drupal.history.markAsRead(' . $node->id() . ');},false);',
-      'type' => 'inline',
+    // Embed the metadata for the "X new comments" link (if any) on this
+    // entity.
+    $build['#post_render_cache']['history_attach_timestamp'] = array(
+      array('node_id' => $node->id()),
     );
-    $build['#attached']['library'][] = 'history/drupal.history';
   }

This is not correct, this has to be triggered by javascript.

The point of doing it in javascript is that it also works when delivering pages from page page/varnish.

The last submitted patch, 12: assets_only_libraries-2382533-12.patch, failed testing.

wim leers’s picture

Status: Needs review » Needs work

#13: good point. I had totally forgotten about that.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new113.13 KB
new4.09 KB

Status: Needs review » Needs work

The last submitted patch, 16: assets_only_libraries-2382533-16.patch, failed testing.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new112.75 KB
new1.86 KB

I'm an idiot. I addressed #13 but then failed to revert the test changes…

Status: Needs review » Needs work

The last submitted patch, 18: assets_only_libraries-2382533-18.patch, failed testing.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new112.74 KB
wim leers’s picture

StatusFileSize
new112.05 KB
new5.24 KB

Self-review time. Plus patch to fix it.

  1. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    + * i.e. tests
    + *
    + * @code
    

    This looks weird.

  2. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    +  /**
    +   * Enable Language and SimpleTest in the test environment.
    +   *
    +   * @var array
    +   */
    +  public static $modules = array('language', 'simpletest', 'common_test', 'system');
    

    {@inheritdoc}

  3. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    +  /**
    +   * Stores configured value for CSS preprocessing.
    +   */
    +  protected $preprocess_css = NULL;
    +
    +  /**
    +   * Stores configured value for JavaScript preprocessing.
    +   */
    +  protected $preprocess_js = NULL;
    ...
    +    // Restore configured value for JavaScript preprocessing.
    +    $config = \Drupal::config('system.performance');
    +    $config->set('js.preprocess', $this->preprocess_js);
    +    $config->save();
    

    This is obsolete, a KernelTestBase's config changes don't persist to the actual site.

  4. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    +    // Disable preprocessing
    

    Missing trailing period.

  5. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    +   * Tests that default JavaScript is empty.
    

    and CSS

  6. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    +  }
    +
    +
    +  /**
    

    Too much whitespace.

  7. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    +   * Tests _drupal_add_js() sets preproccess to FALSE when cache is also FALSE.
    

    s/preproccess/preprocess/

  8. +++ b/core/modules/system/src/Tests/Common/AttachedAssetsTest.php
    @@ -0,0 +1,434 @@
    +   * Tests that multiple modules can implement the same library.
    

    s/the same library/libraries with the same name/

  9. +++ b/core/modules/system/theme.api.php
    @@ -286,18 +286,23 @@
    + *  * Libraries, JavaScript settings, feeds, HTML <head> tags and HTML <head> links
    ...
    + * values are the attached data. For example:
    +
    + *
    ...
    + * values.Example:
    

    Formatting problems.

  10. +++ b/core/tests/Drupal/Tests/Core/Asset/LibraryDiscoveryParserTest.php
    @@ -179,6 +179,37 @@ public function testBuildByExtensionWithMissingInformation() {
    +    $this->assertEquals(\Drupal::VERSION, $libraries['core-versioned']['js'][0]['version']);
    +
    +  }
    

    Whitespace problem.

nod_’s picture

Status: Needs review » Needs work

In common.inc there is still

  // Convert every JavaScript settings asset into a regular JavaScript asset.
  // @todo Clean this up in https://www.drupal.org/node/2382533

What's the plan for that clean-up?

in _drupal_add_library:

          // If the value is not an array, it's a filename and passed as first
          // (and only) argument.
          if (!is_array($options)) {
            $data = $options;
            $options = NULL;
          }
          // In some cases, the first parameter ($data) is an array. Arrays can't be
          // passed as keys in PHP, so we have to get $data from the value array.
          if (is_numeric($data)) {
            $data = $options['data'];
            unset($options['data']);
          }

Seems to me that $data will always be numeric and $options will always be an array because we declare everything through libraries and in the YML, js and css are always objects, not a mix of [filename, filename] and {filename: options, filename: options}:

  js: 
    somefilename.js: {}

Tried it and looks like the following is enough:

      // Add both the JavaScript and the CSS.
      // The parameters for _drupal_add_js() and _drupal_add_css() require special
      // handling.
      foreach (array('js', 'css') as $type) {
        foreach ($library[$type] as $data => $options) {
          call_user_func('_drupal_add_' . $type, $options['data'], $options);
        }
        unset($elements['#attached'][$type]);

in JsCollectionRenderer

    // For inline JavaScript to validate as XHTML, all JavaScript containing
    // XHTML needs to be wrapped in CDATA. To make that backwards compatible
    // with HTML 4, we need to comment out the CDATA-tag.
    $embed_prefix = "\n<!--//--><![CDATA[//><!--\n";
    $embed_suffix = "\n//--><!]]>\n";

I so want those gone, but looks like we still "need" it for drupalSettings. Out of scope though.

In History module
$build['#attached']['drupalSettings']['history']['nodesToMarkAsRead'][$node->id()] = TRUE;
can we have
$build['#attached']['drupalSettings']['history']['nodesToMarkAsRead'][] = $node->id();
Instead? It's the ajaxPageState situation again, are we really worried about duplicates here?

Looks good to me otherwise :)

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new111.75 KB
new1.63 KB
  // Convert every JavaScript settings asset into a regular JavaScript asset.
  // @todo Clean this up in https://www.drupal.org/node/2382533

What's the plan for that clean-up?

Good question. Back then, I thought this issue would be a great place to finally remove _drupal_add_(css|js)(). Turns out this isn't a great place: the patch is already >110 KB because of the essential test coverage updates/fixes.
Now #2368797: Optimize ajaxPageState to keep Drupal 8 sites fast on high-latency networks, prevent CSS/JS aggregation from taking down sites and use HTTP GET for AJAX requests seems like the best place to do that. So updated the todo to link to that issue instead.

Seems to me that $data will always be numeric and $options will always be an array because […]

Great catch! :) I didn't bother fully simplifying here (I copied it *verbatim*), because the big simplification will happen in #2368797: Optimize ajaxPageState to keep Drupal 8 sites fast on high-latency networks, prevent CSS/JS aggregation from taking down sites and use HTTP GET for AJAX requests, where _drupal_add_(css|js|library)() will go away. But if we can do this now already, that'll make the next patch simpler to review, so +1, made that change.

In JsCollectionRenderer […] I so want those gone […] Out of scope though.

+1, and indeed out of scope.

Can we have […] instead? It's the ajaxPageState situation again, are we really worried about duplicates here?

Yes, we are worried about duplicates and hence other drupalSettings overwriting these. It's up to us to provide unique keys so that some other attached drupalSettings don't overwrite ours.

Status: Needs review » Needs work

The last submitted patch, 23: assets_only_libraries-2382533-23.patch, failed testing.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new111.77 KB
new665 bytes

Oops.

Status: Needs review » Needs work

The last submitted patch, 25: assets_only_libraries-2382533-25.patch, failed testing.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new110.41 KB
new1.47 KB

Turns out that this statement:
Seems to me that $data will always be numeric and $options will always be an array because we declare everything through libraries and in the YML
was not entirely true. Quick Edit module has a hook_library_alter() for which this is not true. Hence the test failures in #25. Since it's the only violation in core, it's easy enough to make it comply as well.

dawehner’s picture

This patch looks really great. It reduces a lot the amount of different ways how to do things: 'css' vs 'library', 'library' vs. 'inline' ...

+++ b/core/includes/common.inc
@@ -1305,10 +1285,6 @@ function drupal_clean_id_identifier($id) {
  *   _drupal_add_js(array('myModule' => array('key' => 'value')), 'setting');

@@ -1694,44 +1662,40 @@ function drupal_merge_attached(array $a, array $b) {
+ * - 'drupalSettings' (JavaScript settings)

Just curious, is this example still valid?

wim leers’s picture

It reduces a lot the amount of different ways how to do things: 'css' vs 'library', 'library' vs. 'inline' ...

Yep, that was the goal: declarativeness, simplicity, consistency.

Just curious, is this example still valid?

Good question. The answer is: yes and no.

nod_’s picture

Status: Needs review » Reviewed & tested by the community

Better tests, better DX, everything still working. No more complains.

Basically the patch changes history module to not use inline JS, do the API removal change and do lots of test improvments.

wim leers’s picture

Issue summary: View changes

Updated IS, added beta evaluation.

catch’s picture

Status: Reviewed & tested by the community » Fixed

Only had nitpicks, fixed on commit:

  1. +++ b/core/includes/common.inc
    @@ -1350,11 +1324,11 @@ function drupal_clean_id_identifier($id) {
    + *   the $data parameter ('file'/'setting'/'external'), or an assocative array.
    

    associative

  2. +++ b/core/includes/common.inc
    @@ -1694,44 +1662,40 @@ function drupal_merge_attached(array $a, array $b) {
    + * is an associative array, where the keys are the the attachment types and the
    

    Not introduced by the patch, just moved around, but 'the the'.

Everything else looks great. Simplifies the system itself and makes things more consistent for themes and modules. Also unblocks both the AJAX pagestate optimization and doing asset generation in separate http requests.

Committed/pushed to 8.0.x, thanks!

  • catch committed 77c613e on 8.0.x
    Issue #2382533 by Wim Leers: Attach assets only via the asset library...
Jeff Burnz’s picture

How do I add a dynamically generated stylesheet that could have any name?

wim leers’s picture

Jeff: Define "dynamically generated"?

  • Dynamically generated once a library is available: hook_library_info_alter() to alter the library you care about and add your randomly named CSS file.
  • Dynamically generated as in at runtime, during a page request: hook_library_alter() to alter the library you care about and add your randomly named CSS. Examples include locale_library_alter() and quickedit_library_alter().

Once #2378737: Consider not executing hook_library_(info)_alter() *per* library, but once for all lands, they'll have the exact same signature, they'll just be executed at different times and with different frequencies.

Jeff Burnz’s picture

One of my modules and a theme can build stylesheets - there is no library. I can never know the name of these files in advance, it depends on the user input.

User saves a form, file is generated and saved to the file system. I save the path/name of file etc and load it at the appropriate time.

How can I load these files?

andypost’s picture

Jeff, you will need to manage libraries for each file

catch’s picture

If there's no library for the file to go in, then a completely new library could be added dynamically once #2358981: Provide a mechanism for dynamic library declarations is in.

wim leers’s picture

Or, for now, you could have a library with a dummy file that is altered away.

Jeff Burnz’s picture

Just be aware of this #2050269: hook_library_info_alter() is not called for themes

I'm going to have a crack at solving my issue with hook_css_alter(), since that is probably the only real option I have for now.

What we really need is #2358981: Provide a mechanism for dynamic library declarations

arla’s picture

I think this change record will have to be updated: drupal_add_css(), drupal_add_js() and drupal_add_library() removed in favor of #attached

Perhaps others as well.

andypost’s picture

+++ b/core/modules/comment/comment.libraries.yml
@@ -30,7 +30,7 @@ drupal.comment-new-indicator:
-    - history/drupal.history
+    - history/api

comment does not depends on history. is this fine?

hass’s picture

Status: Fixed » Needs work

From case intro this issue may the the root cause of #2391025: Add support for inline JS/CSS with #attached

I read Remove support for inline CSS and JS assets (see #5.3 + #8 for details). what is a clear failure!

Jeff Burnz’s picture

Heres one of my cases that is, as I see it, very difficult to work around since this change.

I used inline JS for live reload, Typekit fonts and other things, this change forced me to use the asynchronous Typekit method which always gives a FOUT and is pretty dam hard to avoid even using their state classes. Inlining the JS is very fast method for Typekit, same goes for Google fonts, if you inline the link its super fast, but I cannot do that now (users select their font in settings), and I have to use their asynchronous JS method which gives a FOUT most of the time and there are no state classes at all.

I'm looking for ways around these issues, I understand the concerns in the issue and there maybe ways to get around the FOUT's etc, but this has made things much more difficult for front enders.

nod_’s picture

See the 'file-inline' type that we could use to fix this. Need to open an issue about it.

hass’s picture

What is 'file-inline' ? I need full inline code without files. The inline code I'm adding may dynamic per user, too.

nod_’s picture

Status: Needs work » Fixed

Sorry was on my phone couldn't find the issue. What I mean is the "inline js" part of this comment #2298551-9: Documentation updates adding/adjusting inline/external JS in themes.

I'm replying in the other issue. I don't see a reason to go back on the principle of this issue. Inline scripts are not considered assets anymore, it's just random HTML bits, that can be added by #markup in the body of the page or with #attached => html_head.

bzrudi71’s picture

I read a lot regarding the latest changes to js/css libraries here, and in all the related issues. I still can't find a solution for my problem. Consider this - I use a field formatter/widget for displaying/editing a map (Google/OSM). This map is displayed on every node. All the js required for the map itself is defined in my module.libraries.yml (igwmap.formatter.js), easy and clear. For the map layers, for example an GPX track overlay, all the required js is stored in a file during node save, so each single nid has a separate js file to load. For now I did this (in FieldFormatter):

      $elements[$delta] = [
        '#markup' => $map_div,
        '#attached' => [
          'library' => ['igwmap/igwmap.formatter.js'],
          'js' => [
            [
              'type' => 'file',
              'data' => $item->static_js,
              'scope' => 'footer',
              'weight' => 1,
              'defer' => FALSE
            ]
          ]
        ],
      ];

where $item->static_js is the full path to the js file for the current node to load, so something like this files/tracks/js/node_1234.js. I do not see how to work around this by the provided examples, so this seems like a regression to me. Also non of the hooks_library_something() seem to do the job. I see why we need these changes and they all make sense to me, but leave me with a problem I have no idea how to get around. Any suggestions welcome :-)

wim leers’s picture

Instead of saving the data to a JS file and loading that file, save the data to a file or the DB and load it as JS settings. Or, store it as JSON, then let your JS load that JSON. Or, since you're using #markup already, include that JSON in the markup.

Plenty of options :)

bzrudi71’s picture

Thanks @Wim! I did it some other way, but you pointed me to the right direction :-)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

yesct’s picture

making tag the more common one so we can delete the lesser used one so it does not show in the autocomplete