Problem/Motivation

CSS and JavaScript aggregation previously generated files inline during HTML page requests. If the file was requested without a PHP request to the page, and didn't exist, there was no way to create it and you'd get unstyled pages or broken JavaScript. This could happen just from a clear of js/css files on the server and then HTML cached in a CDN.

However it also had to periodically delete css and js files to avoid a potentially infinite collection of files.

To get around this only files older than 30 days are deleted when there's a cache clear.

However, following #1014086: Stampedes and cold cache performance issues with css/js aggregation, as long as a file is valid for libraries available on the site, we can recreate it independent of the 'parent' HTML request, and don't have to worry about the race condition.

Additionally, we store all the created files in state, this is necessary because the asset filename is based on the content of the files themselves, and that would be too expensive to recalculate. The asset filename with the new system is cheaper to recreate, so we don't really need this extra level of i/o caching there at all.

Steps to reproduce

Proposed resolution

Remove the css.cache_files and js.cache_files state entries.

Remove the system.performance stale file threshold.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Comments

catch created an issue. See original summary.

catch’s picture

Title: Consider removing the aggregate stale file threshold » Consider removing the aggregate stale file threshold and state entry
Issue summary: View changes
catch’s picture

Status: Active » Needs review
StatusFileSize
new9.16 KB
catch’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll
ravi.shankar’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
StatusFileSize
new9.16 KB
new2.23 KB

Added reroll of patch #3 on Drupal 10.1.x.

catch’s picture

StatusFileSize
new1.29 KB
new9.18 KB
catch’s picture

Title: Consider removing the aggregate stale file threshold and state entry » Remove the aggregate stale file threshold and state entry

Note that the most recent patch on #2258313: Add license information to aggregated assets currently relies on this entry, but I think it's doing so unnecessarily. Even if similar information is needed, then it would make more sense for that issue to track it explicitly.

catch’s picture

StatusFileSize
new717 bytes
new9.19 KB
anchal_gupta’s picture

StatusFileSize
new0 bytes
new760 bytes

Please Ignore this patch. By mistake blank patch has been uploaded. I will again upload in the same thread

anchal_gupta’s picture

StatusFileSize
new9.34 KB

Added reroll of patch #8 on Drupal 10.1.x.

catch’s picture

StatusFileSize
new605 bytes
new9.34 KB

trailing space.

Status: Needs review » Needs work

The last submitted patch, 11: 3301573-11.patch, failed testing. View results

catch’s picture

Status: Needs work » Needs review
StatusFileSize
new1.4 KB
new10.46 KB
catch’s picture

StatusFileSize
new523 bytes
new10.46 KB

The last submitted patch, 13: 3301573-12.patch, failed testing. View results

nod_’s picture

Status: Needs review » Reviewed & tested by the community

Looks good, update hook as well. Looks like it wasn't tested so better to not have the code around :)

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs reroll

Needs a bit of re-roll due to changes to system.post_update.php. I'd handle that on commit but I've got a question. Given we've only deprecated \Drupal\Core\Asset\CssCollectionOptimizer and \Drupal\Core\Asset\JsCollectionOptimizer and they both still use these state entries shouldn't we wait until Drupal 11 for this?

catch’s picture

@alexpott hmm, that might be reason to delay the update to Drupal 11 (although those classes would just add the state entry back if they were being used, and then it would be cruft), however all the changes apart from the update would need to land in Drupal 10.1 so we can remove the new deprecations in Drupal 11.

ravi.shankar’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
StatusFileSize
new10.42 KB
new2.27 KB

Added reroll of patch #14 on Drupal 10.1.x, and removed needs reroll tag.

Status: Needs review » Needs work

The last submitted patch, 19: 3301573-19.patch, failed testing. View results

catch’s picture

Status: Needs work » Needs review

Random failure.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new85 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

ameymudras’s picture

Status: Needs work » Needs review
StatusFileSize
new10.26 KB

Rerolled the patch

smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs Review Queue Initiative

Running PHPStan on *all* files.
------ ---------------------------------------------------------------------------
Line core/tests/Drupal/Tests/Core/Asset/CssCollectionOptimizerLazyUnitTest.php
------ ---------------------------------------------------------------------------
55 Class Drupal\Core\Asset\CssCollectionOptimizerLazy constructor
invoked with 11 parameters, 10 required.
115 Class Drupal\Core\Asset\CssCollectionOptimizerLazy constructor
invoked with 11 parameters, 10 required.

mrinalini9’s picture

Status: Needs work » Needs review
StatusFileSize
new13.42 KB
new3.04 KB

Updated patch #23 by addressing #24, please review it.

Thanks!

catch’s picture

Status: Needs review » Needs work

The interdiff looks good.

The last thing here is to deal with #17 and #18, I think we should probably remove the post update hook from this patch, and open a follow-up against Drupal 11 to add it there - can just have that part of the patch attached. Marking needs work for that.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

catch credited acbramley.

catch credited Chi.

catch credited lauriii.

catch’s picture

Priority: Normal » Major
Status: Needs work » Needs review
Related issues: +#3370828: Ensure that edge caches are busted on deployments for css/js aggregates
StatusFileSize
new13.54 KB

Marked #3370828: Ensure that edge caches are busted on deployments for css/js aggregates as duplicate.

Re-rolled. Thinking about #17/#18 more I think if someone somewhere really is using the old classes, they should be responsible for cleaning up the state entries too, so leaving the update in. Crediting people from the other issue.

A workaround if you wanted this in 10.0.x would be to set stale_file_threshold to 0.

Bumping to major since this is a DX issue for local development, shame it stalled.

catch’s picture

StatusFileSize
new13.57 KB

Fixing cs issues.

We could also backport the ::deleteAll() changes to 10.0 without the deprecations. That would improve things for local development and deployments.

lauriii’s picture

Status: Needs review » Reviewed & tested by the community

To me #32 sounds reasonable. Even if @alexpott disagrees, it seems fine to run the post update now and we could have another post update later to clean it up again if he feels strongly about this. I don't think there are any usages in contrib.

+++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
@@ -135,27 +131,15 @@ public function optimize(array $css_assets, array $libraries) {
   public function getAll() {

Went through contrib uses of this class and didn't see any calls for this. Seems fine to make this change and deprecate the method.

+1 for backporting the changes to \Drupal\Core\Asset\CssCollectionOptimizerLazy::deleteAll to 10.1.x.

catch’s picture

StatusFileSize
new1.96 KB

Here's the 10.1 backport.

andypost’s picture

+++ b/core/lib/Drupal/Core/Asset/AssetCollectionOptimizerInterface.php
@@ -25,6 +25,11 @@ public function optimize(array $assets, array $libraries);
+   * @deprecated in drupal:10.2.0 and is removed from drupal:11.1.0. There is

+++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
@@ -135,27 +131,15 @@ public function optimize(array $css_assets, array $libraries) {
+    @trigger_error(__METHOD__ . ' is deprecated in drupal:10.2.0 and will be removed in drupal:11.0.0, there is no replacement. See https:// www.drupal.org/node/3301744', E_USER_DEPRECATED);

+++ b/core/lib/Drupal/Core/Asset/JsCollectionOptimizerLazy.php
@@ -150,26 +146,15 @@ public function optimize(array $js_assets, array $libraries) {
+    @trigger_error(__METHOD__ . ' is deprecated in drupal:10.2.0 and will be removed in drupal:11.0.0, there is no replacement. See https:// www.drupal.org/node/3301744', E_USER_DEPRECATED);

Could be fixed on commit - s/11.1/11.0

mherchel’s picture

Will this fix the issue in 10.1 where if a user clears cache, it doesn't refresh the aggregate CSS/JS that's pulled down?

nsciacca’s picture

@mherchel - I just applied this to my 10.1 and other than a warning "] rmdir(/files/css): Directory not empty FileSystem.php:267" it did appear to fix the aggregate CSS/JS on my Pantheon deploy.

catch’s picture

@mherchel yes it should.

mherchel’s picture

@mherchel yes it should.

Just tested this out, and it doesn't appear to be working.

Not sure if this blocks this issue, or if this should be reported in a followup.

Testing steps

  1. Ensure aggregated CSS/JS, caching enabled etc.
  2. View source. Copy filename including querystring and save it (It looks something like

    /sites/default/files/css/css_hegpbZvWLCegz1eocO6sYJ0X_AGboa15OU4bOdIZPaI.css?delta=0&language=en&theme=olivero&include=eJx9UFuSgyAQvBDqGfYkqQF6lV1kqBmI6-0XEzWVVCo_MPQDmnYsGLzUTLHPwqNA1bgN5ORwnyKT_9nRXephud4FqeCvVIoH9UC6GNKvGl21YB4sKZ7uIFfCFTeR4dhG4SHzAoHv7NrZyO5BfAPeXAMWHW5rP7OvESdfaNTzkOgaRiqBU6doeTzJepIKEjd1S_B4Z8gS5jfyRCK8PHcQNEdqHRTmaEmGfTdVIYdom_vQMuz9hVaOpEbohDxBfHPXzVql07K2MsYH8vRcIWsbO1OiEfK5-COJTizF1XJIjrMhP4d0eQneFwE-UP3ErZLX7_ZQRxlfm83YMF5yyBiO4R-XNOx7

  3. Clear cache
  4. View source again and compare the aggregate filename/querystring against what I checked before.
  5. I expect the filename or querystring to change but it doesn't. I also manually modified Olivero's CSS (to intentionally make it out of date), but that had no effect.
catch’s picture

@mherchel can you check if drush cr deletes the aggregate file from disk? And can you try a hard browser refresh to see if you get the new contents from it once it's recreated?

We might want to add the css_js_query_string back to libary URLs (as a separate query parameter, not part of the hash) to ensure the new version gets loaded, but would be its own issue.

mherchel’s picture

Yep. Verified its working 🙌

Testing steps

  1. Verified path of CSS aggregate file
  2. Ran drush cr
  3. noted that entire CSS directory is now removed from sites/files
  4. modified a CSS file in Olivero
  5. Refreshed the path to the CSS file
  6. Noted that the new file is created, and it has the updated content.

Thank you!

lauriii’s picture

+++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
@@ -142,20 +142,7 @@ public function getAll() {
-    $this->state->delete('drupal_css_cache_files');

Shouldn't we keep this?

catch’s picture

@lauriii it's no longer set by anything non-deprecated in core so it's redundant code in the new classes.

catch’s picture

StatusFileSize
new1.01 KB
new1.92 KB

Discussed #43/#44 with @lauriii - while the delete should be a no-op, in the interests of keeping changes to the absolute minimum in 10.1, adding it back.

  • lauriii committed 405f8b33 on 11.x
    Issue #3301573 by catch, ravi.shankar, mrinalini9, mherchel, lauriii,...

  • lauriii committed f2933175 on 10.1.x
    Issue #3301573 by catch, ravi.shankar, mrinalini9, mherchel, lauriii,...
lauriii’s picture

Status: Reviewed & tested by the community » Fixed

Committed 405f8b33a7 and pushed to 11.0.x. Also committed the 10.1.x patch. Thanks!

badwebcoder’s picture

Version: 11.x-dev » 10.1.x-dev
Category: Task » Bug report
Priority: Major » Normal
Status: Fixed » Active

#45, It works when I clear the cache via the admin toolbar, but not when I do it via Drush.
When I run drush cr I get the following messages:

[notice] The file assets://css was not deleted because it does not exist.
[notice] The file assets://js was not deleted because it does not exist.

CSS and JS are not removed and new files are not created.

badwebcoder’s picture

#45, works when I run cache clear via admin toolbar, but doesn't work via Drush.
When I run drush cr command I get the following messages:

[notice] The file assets://css was not deleted because it does not exist.
[notice] The file assets://js was not deleted because it does not exist.

As a result, old CSS and JS files are not deleted, and new ones are not created.

Sorry for the duplicate comment, i accidentally

catch’s picture

Category: Bug report » Task
Status: Active » Fixed

@BadWebCoder please open a new issue. The existing code was already deleting files in that directory, just not the directory itself, so if drush isn't able to find the directory now, it probably wasn't able to find it beforehand either.

chike’s picture

I did get the notices @BadWebCoder mentioned when I cleared cache with Drush,

$ vendor/bin/drush cr
 [notice] The file assets://css was not deleted because it does not exist.
 [notice] The file assets://js was not deleted because it does not exist.

But when I refreshed the page, my CSS changes did apply to the page.

Off-topic: I can also see that some of the forum topics I added on the site which I saw on the PC (it is a fresh install), viewing the site on a phone I can't see the most recent forum topics. Content this time and not CSS or JS.

chike’s picture

Kindly ignore the off-topic above. It was my fault. I had wrong forum access permissions set.

chike’s picture

If CSS aggregation is turned on, CSS changes don't show up on the site and the site remains in the past never changing, whether caches are cleared in Drush or UI.

chi’s picture

The existing code was already deleting files in that directory, just not the directory itself

@catch, seems it is deleting the directory as well.

https://git.drupalcode.org/project/drupal/-/blob/10.1.1/core/lib/Drupal/...

finex’s picture

Currently it's also deleted the JS directory and both are never recreated (tested on both 10.1.1 and 10.1.x-dev).

catch’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

edmund.dunn’s picture

We are having the same issue as @FiNeX in #56.

quietone’s picture

published the change record

wim leers’s picture

AFAICT this incorrectly removed system.performance:stale_file_threshold from config schema. It should have deprecated that instead? 🤔 (That is possible since #2997100: Introduce a way to deprecate config schemas.)