We are in the process of switching to using the s3fs module to host our files. Per s3fs instructions, we added the following to our nginx config:

For nginx, add this to your server config:

location ~* ^/(s3fs-css|s3fs-js)/(.*) {
  set $s3_base_path 'YOUR-BUCKET.s3.amazonaws.com/s3fs-public';
  set $file_path $2;

  resolver 8.8.4.4 8.8.8.8 valid=300s;
  resolver_timeout 10s;

  proxy_pass http://$s3_base_path/$file_path;
}

Technically, we are serving our files through Cloudfront, so I assume the $s3_base_path needs to be the Cloudfront URL. We've tested it both ways, have done a full advagg file and cache clears, and none of the JavaScript files want to load. In the console, I get the following for each of the files:

Loading failed for the <script> with source “http://test-site.com/s3fs-js/advagg_js/js__PLnzmAtDp0j2y5mxKCtH9idVuyqlLjUYuXm-7I6z6N4__bpF254AanmlVOxLRkzbCaSxGD-piSkOP10t_J0Ui4Rw__dW3csPqwDiyfjzaQBYtKa5FrQ9CrBXcl_iUZzmKGRTo.js”.

If I take the actual file URL and put it into a browser, some of the files load as expected, and others give an XML "Access Denied" message (which would tell me it's either not there, or the code is attempting to access a file directly from S3, which has no public access).

Also getting async error messages to flush the cache:

Adv CSS/JS Agg - Async Mode	Flush your caches.
You need to flush your menu cache. This can be done at the top of the performance page; under "Clear cache" press the "Clear all caches" button.

However, no matter how many times the cache is flushed, it makes no difference.

Any thoughts, or additional information I can provide? Thank you for your assistance.

Comments

ron_s created an issue. See original summary.

sgdev’s picture

This seems odd... there are AdvAgg css and js files being created and stored in S3, but the ones being requested through the browser are not the same files. In fact, the ones in the browser HTML don't exist at all, and that is why they are returning "Access Denied."

However, if I attempt to load via URL some of the AdvAgg css and js files that are in S3, they load without issue.

sgdev’s picture

I've found cause of the "Async Mode" error message, and have posted a separate thread and patch: https://www.drupal.org/project/advagg/issues/2972760

Please review, thanks. Continuing to investigate the rest of the issues.

sgdev’s picture

Title: JavaScript files not loaded when accessed via s3fs » Prevent more than 4095 CSS selectors not working with s3fs
Category: Support request » Bug report

While I think there may be some other possible issues, one key problem I've uncovered is the option to "Prevent more than 4095 CSS selectors in an aggregated CSS file" is not working correctly when the s3fs module is being used.

When loading a version of the site not using s3fs, the Bundler and CSS Compression tabs have references to "sites/default/files/advagg_css/parts/sites/all/themes/our_theme/css". These are included in the AdvAgg-generated files.

When using S3, the AdvAgg code has definitely created the parts, and stored them in public://advagg_css/parts/sites/all/themes/our_theme/css (we are using the s3fs takeover of "public://" rather than the "s3://" scheme). However, they are not referenced in Bundler or CSS Compression, and pages are not loading correctly because theme CSS is loading out of order from CSS included in the bundled files.

If I clear all AdvAgg caches and files, the parts files are rebuilt and stored in S3, but they are never referenced in the AdvAgg configuration.

Going to see if I can uncover any other details since this is a fairly serious issue for us.

sgdev’s picture

Status: Active » Needs review
StatusFileSize
new1.57 KB

Ok, I have a patch for review. This seems to fix the problem.

The file is stored in S3 correctly, but it is referenced in the database as a relative path. Whenever the CSS is added to the page, it is never bundled -- always added as an @import.

By adding this adjustment to the filename path to include the scheme for s3fs, it is compressed and bundled as expected.

sgdev’s picture

Just for further clarity, the s3fs module has its own dirname function if you prefer to have the conditional be on the line of code where drupal_dirname is currently called. From the comments:

  /**
   * Gets the name of the parent directory of a given path.
   *
   * This method is usually accessed through drupal_dirname(), which wraps
   * around the normal PHP dirname() function, since it doesn't support stream
   * wrappers.
   *
   * @param string $uri
   *   An optional URI.
   *
   * @return string
   *   The directory name, or FALSE if not applicable.
   *
   * @see drupal_dirname()
   */
  public function dirname($uri = NULL) {
sgdev’s picture

Title: Prevent more than 4095 CSS selectors not working with s3fs » Add consistent support for the s3fs no_rewrite_cssjs setting
Related issues: +#2956531: CKEditor only loads when Deferred JavaScript Execution set to disabled
StatusFileSize
new14.88 KB

I've really dug deep into this, and realized there is a much bigger issue that needs to be addressed. It also ties together several issues I recently posted into a single patch.

The core problem is a lack of consistent support for the s3fs no_rewrite_cssjs setting. This is referenced in several places in the code, but is not fully built out to support every case.

Not only does this cause a problem by not allowing the 4095 CSS selectors, it also causes issues for advagg_relocate. The discussion we had last month (https://www.drupal.org/project/advagg/issues/2956531) will not work when using s3fs, because the relocated files don't even show up as options in the JS Compression settings.

There are also other places in the code that indirectly reference no_rewrite_cssjs that I have included as part of a single, consistent approach. The attached patch should fix all these problems. I'm open to adjusting the patch if you think there is a better way to address.

I looked into modifying the $filename value set for the $record in advagg_insert_update_files, right before the file is saved. This would allow the patch to be a lot smaller. However, given that the path changes per file (public://advagg_css, public://advagg_relocate, etc.), I wasn't sure if there is an easy way to do this.

Maybe if the file uri was passed into $file_meta_data for each file via advagg_get_info_on_files? I'd be interested to know your thoughts, thanks.

sgdev’s picture

StatusFileSize
new12.35 KB

As mentioned in #2972760, after further testing I realized the "Async Mode" issue should not be part of this patch. Have created a new version that removes the code found with that issue.

See attached.

  • mikeytown2 committed e25ec09 on 7.x-2.x authored by ron_s
    Issue #2972528 by ron_s: Add consistent support for the s3fs...
mikeytown2’s picture

Status: Needs review » Fixed

Thanks for all your work; slowing going through all the patches.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.