We are in the process of switching to using the s3fs module to host our files. Per s3fs instructions, we added the following to our nginx config:
For nginx, add this to your server config:
location ~* ^/(s3fs-css|s3fs-js)/(.*) {
set $s3_base_path 'YOUR-BUCKET.s3.amazonaws.com/s3fs-public';
set $file_path $2;
resolver 8.8.4.4 8.8.8.8 valid=300s;
resolver_timeout 10s;
proxy_pass http://$s3_base_path/$file_path;
}
Technically, we are serving our files through Cloudfront, so I assume the $s3_base_path needs to be the Cloudfront URL. We've tested it both ways, have done a full advagg file and cache clears, and none of the JavaScript files want to load. In the console, I get the following for each of the files:
Loading failed for the <script> with source “http://test-site.com/s3fs-js/advagg_js/js__PLnzmAtDp0j2y5mxKCtH9idVuyqlLjUYuXm-7I6z6N4__bpF254AanmlVOxLRkzbCaSxGD-piSkOP10t_J0Ui4Rw__dW3csPqwDiyfjzaQBYtKa5FrQ9CrBXcl_iUZzmKGRTo.js”.
If I take the actual file URL and put it into a browser, some of the files load as expected, and others give an XML "Access Denied" message (which would tell me it's either not there, or the code is attempting to access a file directly from S3, which has no public access).
Also getting async error messages to flush the cache:
Adv CSS/JS Agg - Async Mode Flush your caches.
You need to flush your menu cache. This can be done at the top of the performance page; under "Clear cache" press the "Clear all caches" button.
However, no matter how many times the cache is flushed, it makes no difference.
Any thoughts, or additional information I can provide? Thank you for your assistance.
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | advagg-s3fs_no_rewrite_cssjs_checks-2972528-8.patch | 12.35 KB | sgdev |
Comments
Comment #2
sgdev commentedThis seems odd... there are AdvAgg css and js files being created and stored in S3, but the ones being requested through the browser are not the same files. In fact, the ones in the browser HTML don't exist at all, and that is why they are returning "Access Denied."
However, if I attempt to load via URL some of the AdvAgg css and js files that are in S3, they load without issue.
Comment #3
sgdev commentedI've found cause of the "Async Mode" error message, and have posted a separate thread and patch: https://www.drupal.org/project/advagg/issues/2972760
Please review, thanks. Continuing to investigate the rest of the issues.
Comment #4
sgdev commentedWhile I think there may be some other possible issues, one key problem I've uncovered is the option to "Prevent more than 4095 CSS selectors in an aggregated CSS file" is not working correctly when the s3fs module is being used.
When loading a version of the site not using s3fs, the Bundler and CSS Compression tabs have references to "sites/default/files/advagg_css/parts/sites/all/themes/our_theme/css". These are included in the AdvAgg-generated files.
When using S3, the AdvAgg code has definitely created the parts, and stored them in public://advagg_css/parts/sites/all/themes/our_theme/css (we are using the s3fs takeover of "public://" rather than the "s3://" scheme). However, they are not referenced in Bundler or CSS Compression, and pages are not loading correctly because theme CSS is loading out of order from CSS included in the bundled files.
If I clear all AdvAgg caches and files, the parts files are rebuilt and stored in S3, but they are never referenced in the AdvAgg configuration.
Going to see if I can uncover any other details since this is a fairly serious issue for us.
Comment #5
sgdev commentedOk, I have a patch for review. This seems to fix the problem.
The file is stored in S3 correctly, but it is referenced in the database as a relative path. Whenever the CSS is added to the page, it is never bundled -- always added as an
@import.By adding this adjustment to the filename path to include the scheme for s3fs, it is compressed and bundled as expected.
Comment #6
sgdev commentedJust for further clarity, the s3fs module has its own
dirnamefunction if you prefer to have the conditional be on the line of code wheredrupal_dirnameis currently called. From the comments:Comment #7
sgdev commentedI've really dug deep into this, and realized there is a much bigger issue that needs to be addressed. It also ties together several issues I recently posted into a single patch.
The core problem is a lack of consistent support for the s3fs
no_rewrite_cssjssetting. This is referenced in several places in the code, but is not fully built out to support every case.Not only does this cause a problem by not allowing the 4095 CSS selectors, it also causes issues for advagg_relocate. The discussion we had last month (https://www.drupal.org/project/advagg/issues/2956531) will not work when using s3fs, because the relocated files don't even show up as options in the JS Compression settings.
There are also other places in the code that indirectly reference
no_rewrite_cssjsthat I have included as part of a single, consistent approach. The attached patch should fix all these problems. I'm open to adjusting the patch if you think there is a better way to address.I looked into modifying the
$filenamevalue set for the$recordinadvagg_insert_update_files, right before the file is saved. This would allow the patch to be a lot smaller. However, given that the path changes per file (public://advagg_css, public://advagg_relocate, etc.), I wasn't sure if there is an easy way to do this.Maybe if the file uri was passed into
$file_meta_datafor each file viaadvagg_get_info_on_files? I'd be interested to know your thoughts, thanks.Comment #8
sgdev commentedAs mentioned in #2972760, after further testing I realized the "Async Mode" issue should not be part of this patch. Have created a new version that removes the code found with that issue.
See attached.
Comment #10
mikeytown2 commentedThanks for all your work; slowing going through all the patches.