Skip generating aggregates that won't be used for stale asset requests

I'm facing the same issue. I don't understand the current logic. If the library doesn't want to be preprocessed, why is it automatically minified by default? In that case it makes no sense in preprocess: false.

I believe the issue lies in the logic implemented in two methods: \Drupal\Core\Asset\JsCollectionOptimizerLazy::optimizeGroup and \Drupal\Core\Asset\JsCollectionOptimizer::optimize.

The \Drupal\Core\Asset\JsCollectionOptimizer::optimize respects asset group preprocess status:

          if (!$js_group['preprocess']) {
            $uri = $js_group['items'][0]['data'];
            $js_assets[$order]['data'] = $uri;
          }

The \Drupal\Core\Asset\JsCollectionOptimizerLazy::optimizeGroup function optimizes the group regardless of any conditions. It doesn't have a condition and simply moves on to the optimization process.

This method should looks like:

  public function optimizeGroup(array $group): string {
    $data = '';
    $current_license = FALSE;

    // No preprocessing, single JS asset: just use the existing URI.
    if ($group['type'] === 'file' && !$group['preprocess']) {
      $data = file_get_contents($group['items'][0]['data']);
    }
    else {
      foreach ($group['items'] as $js_asset) {
        // Ensure license information is available as a comment after
        // optimization.
        if ($js_asset['license'] !== $current_license) {
          $data .= "/* @license " . $js_asset['license']['name'] . " " . $js_asset['license']['url'] . " */\n";
        }
        $current_license = $js_asset['license'];
        // Optimize this JS file, but only if it's not yet minified.
        if (isset($js_asset['minified']) && $js_asset['minified']) {
          $data .= file_get_contents($js_asset['data']);
        }
        else {
          $data .= $this->optimizer->optimize($js_asset);
        }
        // Append a ';' and a newline after each JS file to prevent them from
        // running together.
        $data .= ";\n";
      }
    }

    // Remove unwanted JS code that causes issues.
    return $this->optimizer->clean($data);
  }

I created the MR with the fix and it works with the google_tag module. However, I'm still not sure about this part of the code: $data = file_get_contents($group['items'][0]['data']);. The same code occurs in the methodDrupal\Core\Asset\JsCollectionOptimizer::optimize. It seems that there may be more than one 'item' in the group, and in such a case, some content may be missing. Need some advice here is that safe or not.

Also, I think it might be a better solution to simply update the condition in the loop of the group, from:

        if (isset($js_asset['minified']) && $js_asset['minified']) {
          $data .= file_get_contents($js_asset['data']);
        }
        else {
          $data .= $this->optimizer->optimize($js_asset);
        }

to

        if (!$group['preprocess'] || (isset($js_asset['minified']) && $js_asset['minified'])) {
          $data .= file_get_contents($js_asset['data']);
        }
        else {
          $data .= $this->optimizer->optimize($js_asset);
        }

This will also preserve license in the file contents and will handle multiple items properly.

It also fails when using libraries with external dependencies, but with a slightly different exception.

Only file JavaScript assets can be optimized.

E.g. of such library:

example:
  version: 1
  remote: https://example.com
  license:
    name: Example
    url: https://example.com/license
    gpl-compatible: true
  js:
    js/init.js: { }
    //example.com/script.js: { preprocess: false }

The suggested solution won't work in this case because the type here is external, which is handled in the method \Drupal\Core\Asset\JsCollectionOptimizer::optimize, but not in the method \Drupal\Core\Asset\JsCollectionOptimizerLazy::optimizeGroup. But for now, I’m out of ideas on how to solve this issue for external libraries. The result of the lazy method should be optimized JavaScript.

For those who have encountered this problem, I suggest explicitly setting the following in the code: {preprocess: false, minified: true} for such JavaScript files.

It's also worth mentioning that the solution from MR and Drupal\Core\Asset\JsCollectionOptimizer::optimize does not include injecting library license information.

Can the issue summary be updated to include any proposed solution and other relevant sections from the issue template.

Previously was tagged for tests may be good to get those written and may help guide the fix.

I can confirm this message keeps on filling our logs on larger projects.

It appears this is also affecting the Stripe module.

We're seeing this too. Its odd that it would skip "minified" but not "preprocess = false" here if the optimize function is just going to throw an exception for "preprocess = false": https://git.drupalcode.org/project/drupal/-/blob/11.x/core/lib/Drupal/Co...

Just reporting in that we are seeing this frequent visitor in our logs too.

Same issue here. A larger project fills the log with this error message.

This indeed still fills up the logs and is confusing. This is nothing that should be at exception level, I think. For us, it means, that monitoring sees this as (kind of critical) error like the application is broken somewhere. We should really fix this, or at least reduce the severity for that reason. Especially because the site owner can't do anything about it?

Same as in #14 - using external js, not minified.
Works for me:
- using pach from MR
- using local copy of the external js file, with {preprocess: false, minified: false}

Could we change the exception to an assert() maybe? This seems like it should be a libraries.yml validation step (which we don't really have).

The MR looks reasonable to me though.

Adjusted the issue summary and replaced the exception statements with "assert()" statements. Do we really need tests for this, since it is pretty straight forward?

Also should we create a follow-up issue, regarding #10:

The suggested solution won't work in this case because the type here is external, which is handled in the method \Drupal\Core\Asset\JsCollectionOptimizer::optimize, but not in the method \Drupal\Core\Asset\JsCollectionOptimizerLazy::optimizeGroup. But for now, I’m out of ideas on how to solve this issue for external libraries. The result of the lazy method should be optimized JavaScript.

?

Seems like a pretty valid bug for test coverage.

But pipeline has issues.

Using a patch from the MR. The last commit:
"ef07f56b - Use asserts instead of if cases"
breaks my site. I had to revert it.

@szato what was the error / backtrace message?

@grevil

The website encountered an unexpected error. Try again later.

AssertionError: Only file JavaScript assets can be optimized. in assert() (line 30 of core/lib/Drupal/Core/Asset/JsOptimizer.php).
Drupal\Core\Asset\JsOptimizer->optimize() (Line: 184)
Drupal\Core\Asset\JsCollectionOptimizerLazy->optimizeGroup() (Line: 185)
Drupal\system\Controller\AssetControllerBase->deliver()
call_user_func_array() (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 638)
Drupal\Core\Render\Renderer->executeInRenderContext() (Line: 121)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext() (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 181)
Symfony\Component\HttpKernel\HttpKernel->handleRaw() (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle() (Line: 53)
Drupal\Core\StackMiddleware\Session->handle() (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle() (Line: 28)
Drupal\Core\StackMiddleware\ContentLength->handle() (Line: 116)
Drupal\page_cache\StackMiddleware\PageCache->pass() (Line: 90)
Drupal\page_cache\StackMiddleware\PageCache->handle() (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle() (Line: 124)
Drupal\cloudflare\CloudFlareMiddleware->handle() (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle() (Line: 36)
Drupal\Core\StackMiddleware\AjaxPageState->handle() (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle() (Line: 741)
Drupal\Core\DrupalKernel->handle() (Line: 19)

Uninstalled cloudflare, same error

The website encountered an unexpected error. Try again later.

AssertionError: Only file JavaScript assets can be optimized. in assert() (line 30 of core/lib/Drupal/Core/Asset/JsOptimizer.php).
Drupal\Core\Asset\JsOptimizer->optimize() (Line: 184)
Drupal\Core\Asset\JsCollectionOptimizerLazy->optimizeGroup() (Line: 185)
Drupal\system\Controller\AssetControllerBase->deliver()
call_user_func_array() (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 638)
Drupal\Core\Render\Renderer->executeInRenderContext() (Line: 121)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext() (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 181)
Symfony\Component\HttpKernel\HttpKernel->handleRaw() (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle() (Line: 53)
Drupal\Core\StackMiddleware\Session->handle() (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle() (Line: 28)
Drupal\Core\StackMiddleware\ContentLength->handle() (Line: 116)
Drupal\page_cache\StackMiddleware\PageCache->pass() (Line: 90)
Drupal\page_cache\StackMiddleware\PageCache->handle() (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle() (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle() (Line: 36)
Drupal\Core\StackMiddleware\AjaxPageState->handle() (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle() (Line: 741)
Drupal\Core\DrupalKernel->handle() (Line: 19)

Thank you, @szato! I am an idiot and forgot to negate the conditions... my apologies. Should work again now!

@grevil,

I can confirm: the actual MR patch works for me (with the last commit fix).

Pipeline is green, maybe the Draft status should be removed from the MR?

Pipeline is green now!

Tested the MR patch. Solved my issue, moving to RTBC.

For external JS, I had to move (make a copy) the external JS to local. I think we need a follow-up issue as it was mentioned in #22.

The change doesn't look right to me - how does a single, non-preprocessed file end up getting run through the optimizer? Only files that end up in asset aggregates can be optimized, which implies 'preprocess: true' (although we should change that to 'aggregate: true' at some point). Feels like it should prevent getting to this point earlier rather than returning the same file from the optimizer.

I do think this needs tests, so at least moving back for that, this might help clarify my confusion above.

We are seeing this error in our logs despite having both preprocess: false and minified: true in our library definition.

Correction: There are contrib modules in our codebase that use preprocess: false without minified: true in their library definitions, so it stands to reason that the culprit likely lies within one or more of those.

Hello, if you are hitting "Only file JavaScript assets can be optimized" or "Only file CSS assets can be optimized" I just created #3535330: Assets paths in CSS no longer rewritten when aggregation is enabled and proposed a possible solution. The MR here did not solve my issue because it looks like this is different case than the one reported here.

Correction: There are contrib modules in our codebase that use preprocess: false without minified: true in their library definitions, so it stands to reason that the culprit likely lies within one or more of those.

I don't think it's wrong to do that.

We're seeing this very frequently now on sites that upgraded to Drupal 11. Still NW for the tests here.

I think adding a test here would be like using the library implementation google_tag:

 gtm:
   header: true
   js:
     js/gtm.js: { preprocess: false }

without minified: true.
See #3413105: Only file JavaScript assets with preprocessing enabled can be optimized.

Without the fix here (or a hard-coded minified: true), we'd expect the exception from the issue summary with JS aggregation (/admin/config/development/performance > Bandwidth optimization > Aggregate JavaScript files) enabled.

With this fix, we'd just expect that { preprocess: false } implicitly disables minification and throws no exception any more.

Alright, I added a test to show the issue. The tests only MR test pipeline fails with the expected "Exception: Only file JavaScript assets with preprocessing enabled can be optimized." exception and the MR with the fix + test is green!

Please review!

Thanks @grevil for the test fixes!
Same test-only tests fail, tests are green. So LGTM!

The test-only tests show the expected result:

    E                                                                   1 / 1 (100%)
    
    Time: 00:00.082, Memory: 8.00 MB
    
    Js Collection Optimizer Lazy Unit (Drupal\Tests\Core\Asset\JsCollectionOptimizerLazyUnit)
     ✘ Optimize group preprocess disabled omit minified
       ┐
       ├ Exception: Only file JavaScript assets with preprocessing enabled can be optimized.
       │
       │ /builds/issue/drupal-3416508/core/lib/Drupal/Core/Asset/JsOptimizer.php:34
       │ /builds/issue/drupal-3416508/core/lib/Drupal/Core/Asset/JsCollectionOptimizerLazy.php:170
       │ /builds/issue/drupal-3416508/core/tests/Drupal/Tests/Core/Asset/JsCollectionOptimizerLazyUnitTest.php:82
       ┴
    
    ERRORS!
    Tests: 1, Assertions: 0, Errors: 1.

This is still filling up logs, so let's protect the environment and proceed here :) 🌳
RTBC from my side!

@smustgrave could you have a look again?

I still have the same question as #33, which I've also asked on the MR. ::optimizeGroup() should only ever run for files with preprocess: true, and there isn't an explanation here on how that ends up running for files with preprocess:false.

The unit test shows that you can get to this code path in a unit test, but it does not show how you can get to it in core + a library definition. The unit test could also test that the exception gets thrown instead.

Are there steps to reproduce that happen only with a module like google tag manager and core? I think this probably needs a functional or kernel test to demonstrate the issue.

If it can't be reproduced with core + a module with a suitable library definition, then it could be a contrib module interfering, maybe advagg or similar?

::optimizeGroup() should only ever run for files with preprocess: true, and there isn't an explanation here on how that ends up running for files with preprocess:false

Where is that defined in code? I can't find a single line of code in core, that suggests, that "optimizeGroup()" should only run for assets with "preprocess" set to false?

The flow is as follows:

The "system.routing.yml" defines "\Drupal\system\Routing\AssetRoutes::routes" as a route callback. In "routes()" "Drupal\system\Controller\JsAssetController::deliver" is used as one of the route controller methods. This simply references the parent method "AssetControllerBase::deliver()". And in there we run $data = $this->optimizer->optimizeGroup($group);, but "deliver()" doesn't have a single direct / indirect check on the "preprocess" property, so I am not really sure where this before mentioned check happens?

Updated the comment accordingly. Please see my comment in #43 regarding the "optimizeGroup" objection.

Where is that defined in code? I can't find a single line of code in core, that suggests, that "optimizeGroup()" should only run for assets with "preprocess" set to false?

The "system.routing.yml" defines "\Drupal\system\Routing\AssetRoutes::routes" as a route callback. In "routes()" "Drupal\system\Controller\JsAssetController::deliver" is used as one of the route controller methods.

Because that route only exists to server aggregated (preprocessed) assets.

Individual files that are not preprocessed are delivered directly via a link to the original file, not the asset route.

We're seeing this on same backtrace as written in #16 so it's definitely happening. I agree, the question is where and why. The issue summary reports this from google_tag: #3413105: Only file JavaScript assets with preprocessing enabled can be optimized.. Maybe it's another contrib module that causes this?

I was already able to find out that it doesn't happen on each page load and not even after clearing caches, so maybe this happens in a race condition or whatever?
The question is: Would it make things worse to also sort out such cases here (once we're sure we understand what's happening). We could then leave a comment in code, why it's necessary (if it is)...

I've added some more logging here in the affected projects, so hopefully we'll know more about it tomorrow.

#46 Maybe you are hitting the same issue I reported in #3536795: "Only file JavaScript/CSS assets can be optimized" errors in logs
This can happen when old aggregates are requested or by manipulating the include or delta parameter (so that drupal will attempt to optimize a type: external group).
I've updated 2 projects from D10.3 to D11.2 and hit this in both cases, but I also have seen it on Drupal 10 alone after making changes to library definitions.

@herved thanks that is useful information.

I think it would be possible to add to the existing validation in AssetControllerBase to catch cases where e.g. an external library gets included in an aggregate.

Or possibly we should be moving $data = $this->optimizer->optimizeGroup($group); inside the

    if (hash_equals($generated_hash, $received_hash)) {

check?

Added a new MR for the approach in #48 https://git.drupalcode.org/project/drupal/-/merge_requests/13743

Thanks @catch. What my debugging says is kind of crazy and would IMHO mean we have a more general bug:

It logged:
Only file JavaScript assets **with preprocessing enabled** can be optimized. File: "modules/contrib/google_tag/js/gtag.ajax.js", Type: "file", Preprocess: FALSE"

but gtag.ajax.js has no flags at all:

gtag.ajax:
  js:
    js/gtag.ajax.js: { }
  dependencies:
    - core/drupal.ajax

Here's the full google_tag.libraries.json:
https://git.drupalcode.org/project/google_tag/-/blob/2.0.x/google_tag.li...

Am I missing something or does that look like there's something wrong with the aggregation in general?

Otherwise, I would agree with #47 that it's not the default library loading case we're talking about here. The aggregate URL is definitely not manipulated (by us) in our case, so and old aggregate would make sense, especially because we're seeing more of this over time.

This location is logged with the error:
https://www.example.com/sites/default/files/js/js_Tn3zCv2BVTTEfmbwa6nvePuocXuQ5CnPChmGfXdWQn0.js?delta=5&include=eJyFkV2WwyAIhTeUxm3MLjio1JAa8YD9mVl9Tdtpmz7MvAD3-ol4QDNqwGWm0ETdbGPTozWKNkk1uAmPMdHgWbEiBFkW0kCALXRR9qwLNpbiwkThsLFgITNMna08-Iw_386zjDjjpUvp9Fq6NcBN2_BqHyM0gYDaPp75j-g95MBk7pFn018LcPvdfs2otC8sMdOL2mMg32uofKHstvJJpYbJraE7SutcVUrvZmNUOWfwaLTbFTwNL-08hkNSOZa4Wygybs-M3nXUY8X8cMLEOb4jFRVTX8lk4M2lLL6zf0-eRFImeM79oe-LqWJtkgSzOS7chhPT2dwt3oEz-b3o4h557L_Ikq6YXeYT&language=de&scope=footer&theme=my_custom_theme

PS: These two files are logged in the project:

modules/contrib/posthog/modules/posthog_js/js/init.js
modules/contrib/google_tag/js/gtag.ajax.js

Here's the posthog_js.libraries.yml:
https://git.drupalcode.org/project/posthog/-/blob/1.x/modules/posthog_js...

Both do NOT have preprocess: false flags set!
And I didn't find anything special about them yet.

Both files have in common that they are dynamically added in hook_page_attachments(). Might that be a reason, perhaps?

One more thing I just found out: All these errors triggered last night were from Google / Microsoft crawlers. This could be a coincidence, but it could also have a specific reason, such as old entries in the search engine cache / index.

Running a simple crawler across the current sites (broken link check e.g.) doesn't seem to trigger the error.

Per day, we get 20-50 of these errors.

@anybody if the URL is stale/invalid and doesn't match anything that the site would produce, then https://git.drupalcode.org/project/drupal/-/merge_requests/13743 ought to prevent this from happening. We do have intentional logic for URLs going stale, because library definitions change over time (but this case does seem weird and not necessarily only due to staleness).

Do you have any modules installed on your site that tweak JavaScript aggregation? Specifically advagg but also any of the other js minification modules that are around? Or any custom hook_js_alter() or hook_library_info_alter() or similar?

@catch thanks! No advagg. But you took me on the right road to demystify #51: https://git.drupalcode.org/project/cookies/-/blob/2.x/modules/cookies_gt...

And probably that's the root cause for the edge-case here, besides stale library urls: We're using COOKiES on both projects and cookies_gtag does the following: https://git.drupalcode.org/project/cookies/-/blob/2.x/modules/cookies_gt...

/**
 * Implements hook_js_alter().
 */
function cookies_gtag_js_alter(&$javascript, AttachedAssetsInterface $assets) {
  $doKo = Drupal::service('cookies.knock_out')->doKnockOut();
  if ($doKo) {
    $module_path = \Drupal::service('extension.list.module')->getPath('google_tag');
    $scripts = [
      'gtag' => $module_path . '/js/gtag.js',
      'gtag_ajax' => $module_path . '/js/gtag.ajax.js',
      'gtm' => $module_path . '/js/gtm.js',
    ];
    foreach ($scripts as $key => $script) {
      if (isset($javascript[$script])) {
        $javascript[$script]['preprocess'] = FALSE;
        $javascript[$script]['attributes']['type'] = CookiesConstants::COOKIES_SCRIPT_KO_TYPE;
        $javascript[$script]['attributes']['id'] = 'cookies_gtag_' . $key;
        $javascript[$script]['attributes']['data-cookieconsent'] = 'gtag';
      }
    }
  }
}

note: $javascript[$script]['preprocess'] = FALSE;

and here we go!!

Klaro (part of Drupal CMS) does similar:
https://git.drupalcode.org/project/klaro/-/blob/3.x/klaro.module#L85

/**
 * Implements hook_js_alter().
 *
 * Handles script files added from libraries.
 */
function klaro_js_alter(&$javascript, AttachedAssetsInterface $assets) {
  // @todo The assets are cached based on theme, library, langcode etc. so we
  // cannot differ based on user permissions.
  // @see \Drupal\Core\Asset\AssetResolver::getJsAssets
  /** @var \Drupal\klaro\Utility\KlaroHelper $helper */
  $helper = \Drupal::service('klaro.helper');

  if (!$helper->hasAccess() || $helper->onDisabledUri() || !$helper->getSettings()->get('auto_decorate_js_alter') || !$helper->consentManagementRequired()) {
    return;
  }

  foreach ($javascript as $path => &$script) {
    if ($script['type'] !== 'setting') {
      $app = $helper->matchKlaroApp($path);
      if ($app) {
        $script['preprocess'] = FALSE;
        $script['klaro'] = $app->id();
      }
    }
  }
}

because both modules need this code to knock-out and identify libraries.

@anybody OK so I think my MR will make core a bit more resilient against this, but this is a bug in both modules, the way they're doing this isn't compatible with how aggregation works, as Klaro's @todo points out.

I think they could probably do something like conditionally attach their own library to the page (with a non-existent file similar to how locale and ckeditor5 does it), then when that library is present, alter the other definitions. Because the library would either be there or not, it will result in unique aggregate URLs that way.

Even though aggregation changed a lot in Drupal 10.1, the caching of hook_js_alter() did not, so it would end up with the wrong results if not the specific error being seen here.

Alright, static patch for the time being. In theory, this could fix our problem, depending on whether "preprocess" is part of the hash.

I am still unsure how we end up in optimizeGroup, for a js library, that has preprocess set to false (even if altered through a third party module (e.g. cookies). But this might do the trick.

@catch thanks!! And I guess core won't be able to handle this in the near future as-is? (I mean, this is also risky for other cases where people might do this).

@grevil could you please create an issue in both modules (if not yet existing in klaro) and link to #58 there?

I'll try your MR (#59 as static patch) in both projects now. If anyone is fine with it, should we RTBC it or how to proceed here best?

PS @catch should we also use this to have a look at #3536795: "Only file JavaScript/CSS assets can be optimized" errors in logs which seems at least heavily related to the discussion before? Think it would make sense to solve both now?

I created the two issues here #3557114: Setting "preprocess" false in "klaro_js_alter" could lead to unintentional problems with file aggregation and here #3557113: Setting "preprocess" false in "cookies_mySubmodule_alter" could lead to unintentional problems with file aggregation.

#3536795: "Only file JavaScript/CSS assets can be optimized" errors in logs might be actually fixed through this approach.

These errors seem to come from the fact that Drupal is getting requests from old aggregates (possibly browser cache or other means?).

If I understand correctly, we would assert the old asset hash with the new one, which would fail and lead to "optimizeGroup" not being called at all, meaning no error would be thrown.

We are definitely seeing less watchdog entries now that we are using the patch! Once the issue on Klaro's / Cookie's end is fixed, they'll probably be gone permanently.

RTBC!

Re-titling to reflect what the actual change is now.

This doesn't prevent the code path from klaro et al, just means it will run less often. I can't really think of a test for this, it's mostly just doing less work.

Committed and pushed cdfabaf11f5 to 11.x and 59721e9881d to 11.3.x and 7ce663c45a7 to 10.6.x. Thanks!

Backported to 10.6.x as a bugfix for contrib.

@catch FYI if you should have any ideas on this: We still ran into this issue if we're using custom attributes on the libraries (e.g. see #3558428: Exception: Error trying to optimize JavaScript asset: modules/contrib/posthog/modules/posthog_js/js/cdn.js). In that example we set a custom id="posthog-cdn" id on the script for example to allow cookie banners to block the scripts.

cdn:
  js:
    js/cdn.js:
      attributes:
        id: posthog-js-cdn

We're lucky because we don't need the IDs any more in that case, so we were able to remove it and get around that issue. But it looks like this could lead to similar issues like preprocess: false. Logically, it makes sense that scripts with custom attributes also need to be printed individually (while they might still get optimized / minified) to apply these attributes. Maybe similar to this one: #1587536: JavaScript aggregation should account for "async" and "defer" attributes
Maybe that's exactly what happens, when setting custom attributes?

Just wanted to let you know early, if you should have any ideas. That will be a follow-up anyway IMHO.

@anybody assets with any attributes should be excluded from aggregation and therefore not end up in any aggregates, the defer/async issue is trying to change this so that when sequential assets have the same attribute they do get aggregated, but it shouldn't have any impact on this issue either way.

I created the "follow-up" issue here: #3560254: Libraries with attributes that are solely used as dependencies for other libraries throw an aggregation error.

@catch note, that we have two libraries having an ID attribute set through their library definition, but only the one, which is solely used as a dependency (and not loaded directly) is having this issue and causing the "Exception: Error trying to optimize JavaScript asset" error.

FWIW, this does not seem to solve the case I encounter in #3536795: "Only file JavaScript/CSS assets can be optimized" errors in logs
I just tried the commit here cdfabaf1, and remove the patch from that issue, hoping to close as duplicate.

When I try one of the URLs hitting prod locally, I see $include_libraries contain only libs that should be aggregated but the $delta = 2, which selects a $group inside $groups (containing 4 groups - indexed 0 to 3).
The hash_equals condition passes.
And here is the selected $group:

Array
(
    [type] => external
    [attributes] => Array
        (
            [defer] => 1
        )
    [group] => -100
    [version] => 1.x
    [cache] => 1
    [preprocess] => 
    [license] => Array
        (
            [name] => GPL-2.0-or-later
            [url] => https://www.drupal.org/licensing/faq
            [gpl-compatible] => 1
        )
    [scope] => footer
    [items] => Array
        (
            [0] => Array
                (
                    [type] => external
                    [attributes] => Array
                        (
                            [defer] => 1
                        )
                    [group] => -100
                    [data] => https://webtools.europa.eu/load.js
                    [version] => 1.x
                    [weight] => 0.0004
                    [cache] => 1
                    [preprocess] => 
                    [license] => Array
                        (
                            [name] => GPL-2.0-or-later
                            [url] => https://www.drupal.org/licensing/faq
                            [gpl-compatible] => 1
                        )
                    [scope] => footer
                )
        )
)

Note: this is from https://github.com/openeuropa/oe_webtools
So I'm confused... not sure there's anything wrong with that module, and what scenarios this issue here fixes?
But I believe we are missing some kind of validation on delta. See MR in #3536795.

@herved can you reproduce this URL being generated or is it purely appearing in logs?

I don't think I can generate these URLs. As mentioned in #47, they are most likely stale asset requests hitting PROD. We've also seen some China-hosted sites (detected via referer) cloning our pages, which could explain why most of these requests spike after deployments, with a few continuing afterward.
Another possibility is that bots are manipulating query strings, but I find the first explanation more likely.

In any case, these requests are problematic because they’re filling up the logs with errors.

One possible thing to do would be to add additional validation of the assets in the group, if any of them shouldn't appear in an aggregate, then throw a BadRequestException.

One possibility based in #75 is maybe a deployment changed nothing about the asset aggregate except the delta, and then the old delta ended up being a file that won't be aggregated at all?

@catch I updated #3536795: "Only file JavaScript/CSS assets can be optimized" errors in logs, added tests and steps to reproduce.