Empty JS on multilingual's default language pages; sometimes binary (gzip) or 504 as well. 307 issue.

Ok so it sounds like I need to add in some checks on the status report page if https://www.drupal.org/project/search404 is enabled.

Another thing to note is I had to do quite a bit of work inside of advagg_url_inbound_alter() due to language hooks thinking it needed to add /en or /de to the URL and thus causing drupal to 404 when it hit the advagg_css or advagg_js path. So that is something to look into if you think this is still an issue.

Changing title to reflect what was committed. Also did some readability improvements to the code, thus the large patch size.

Updated title again.

The nginx settings are mainly for far-future headers and gzip handling, things that shouldn't cause an issue like you're describing. The status report will complain about it, but it shouldn't cause empty js files.

Next time it happens can you capture the source html? Maybe there's something in there.

Looking into this more and I think it has to do with 307s in the advagg code.

What was committed. Also threw in a couple of other fixes.

Are there any watchdog messages under advagg that say "Write to file system failed"? After writing to the disk I do a check to see if the file exists and is not zero bytes, advagg will remove the file from the filesystem if it is a zero byte file and choose to always serve the file from PHP until the file exists and the filesize is not zero. See advagg_save_aggregate().

In your case, what's more interesting is the fact that the file names listed in the html document for css & js do not match the pattern that advagg uses. Inside the IE conditional html for CSS you can see that the advagg file structure is used.

<!--[if IE 9]>
<link type="text/css" rel="stylesheet" href="/sites/default/files/advagg_css/css__5xTuj2mwu7YCM5tdqJCSTqb8IFaJsuZtBdlkhPWAvtI__zECT2EO-mYfEcAdyaoiuFULID5G4LnjbTyTBuPvoOcE__cO_Q8asjpbDxPaTmZsGOUZ1ZNBjsRlmOEAPGPY4bGIQ.css" media="all" />
<![endif]-->

We have 3 groups of 43 base64 characters generated by drupal_hash_base64().

5xTuj2mwu7YCM5tdqJCSTqb8IFaJsuZtBdlkhPWAvtI
zECT2EO-mYfEcAdyaoiuFULID5G4LnjbTyTBuPvoOcE
cO_Q8asjpbDxPaTmZsGOUZ1ZNBjsRlmOEAPGPY4bGIQ

1st group is a hash of the files used; 2nd is a hash of the files contents (used for far-future versioning); 3 is a hash of the settings used to generate the files. With the 1st and 3rd data points I can re-generate the file as it was intended to be created.

But for the main CSS you have this structure

<link type="text/css" rel="stylesheet" href="studie-Dateien/css__N3Qa2SmcuP7F06cOB-qAUg2J-lYifmvizNvYl8-OAIg__jQl4fyZi1M.css" media="all">

Looks like the first group of 43 is preserved but after that it doesn't match at all.

N3Qa2SmcuP7F06cOB-qAUg2J-lYifmvizNvYl8-OAIg
jQl4fyZi1M
???

It's also not in the advagg directory.

Something above Drupal is intercepting the HTML, scanning for CSS & JS files and rewriting them on the fly. Code for the JS

<script src="studie-Dateien/js__EUF5gK55x70ZwlK-s8-fYLZZPRk-tnNquiNr6ZUnRxs__l9kxUZWt1PpJ.js"></script>

Has the same structure as the CSS above

EUF5gK55x70ZwlK-s8-fYLZZPRk-tnNquiNr6ZUnRxs
l9kxUZWt1PpJ
???

AdvAgg can not be blamed for the issue you're having because advagg is not being used in the HTML.

Any news on this?

Tentatively re-opening this as we have recently switched on Advanced Aggregation on a high-traffic site hosted at Pantheon, and are running into the same issue described here: occasionally, the some of the js files generated by advagg are 0 bytes (on disk).

• We're seeing empty js files for different sets of source js files
• There are no advanced aggregation watchdog errors in the last couple of days, nor are there any errors/warnings in watchdog around the time of creation

For example:
Nginx access log entry:

10.223.192.191 - - [17/Dec/2015:00:57:01 +0000]  "GET /sites/default/files/advagg_js/js__ZuBmvscliaCjMOn9xLUXCuMFQZIWLK-xK479mu_9GWQ__HcYUy370g4g3kXMcthBj9ZH4LzmuRjdF1Qs7PMiGRjE__FEmgguGjeV4P05hgIH0yZtC5S61T1qSZis52FTUlYEU.js HTTP/1.1" 200 0 "-" "Amazon CloudFront" 0.000 "24.7.29.209, 54.239.203.106, 127.0.0.1, 127.0.0.1”

AdvAgg detailed info:

Array
(
    [0] => js
    [1] => ZuBmvscliaCjMOn9xLUXCuMFQZIWLK-xK479mu_9GWQ
    [2] => HcYUy370g4g3kXMcthBj9ZH4LzmuRjdF1Qs7PMiGRjE
    [3] => Array
        (
            [variables] => Array
                (
                    [advagg_gzip] => 1
                    [is_https] => 1
                    [advagg_global_counter] => 3
                    [base_path] => /
                    [advagg_ie_css_selector_limiter] => 
                    [advagg_ie_css_selector_limiter_value] => 4095
                    [advagg_scripts_scope_anywhere] => 
                    [advagg_devel] => 
                    [advagg_convert_absolute_to_relative_path] => 
                    [advagg_convert_absolute_to_protocol_relative_path] => 1
                    [advagg_force_https_path] => 
                    [advagg_css_dir] => public://advagg_css
                    [advagg_js_dir] => public://advagg_js
                    [language] => en-BE
                    [cdn_mode] => basic
                    [cdn_farfuture_status] => 0
                    [cdn_https_support] => 1
                    [cdn_status] => 2
                    [cdn_request_is_https] => 1
                    [cdn_check_drupal_path] => 1
                )

            [hooks] => Array
                (
                    [advagg_get_css_file_contents_pre_alter] => Array
                        (
                        )

                    [advagg_get_css_file_contents_alter] => Array
                        (
                        )

                    [advagg_get_css_aggregate_contents_alter] => Array
                        (
                        )

                    [advagg_get_js_file_contents_alter] => Array
                        (
                        )

                    [advagg_get_js_aggregate_contents_alter] => Array
                        (
                        )

                    [advagg_save_aggregate_pre_alter] => Array
                        (
                        )

                    [advagg_save_aggregate_alter] => Array
                        (
                            [0] => advagg
                        )

                    [advagg_current_hooks_hash_array_alter] => Array
                        (
                        )

                    [advagg_get_root_files_dir_alter] => Array
                        (
                        )

                    [advagg_context_alter] => Array
                        (
                            [0] => advagg
                        )

                )

        )

    [files] => Array
        (
            [sites/all/modules/contrib/jquery_update/replace/ui/external/jquery.cookie.js] => Array
                (
                    [filesize] => 3655
                    [mtime] => 1450276723
                    [filename_hash] => klaD6918zJ9f6wr59c7bd2RiEZ5F-6lyEP5Mn1v9u6k
                    [content_hash] => S6A-VyA-pXjsUfVtMXppzCu4OvCTN4BoOJD9ngRrZuU
                    [linecount] => 89
                    [data] => sites/all/modules/contrib/jquery_update/replace/ui/external/jquery.cookie.js
                    [fileext] => js
                    [cache_id] => advagg:file:klaD6918zJ9f6wr59c7bd2RiEZ5F-6lyEP5Mn1v9u6k
                )

            [modules/contextual/contextual.js] => Array
                (
                    [filesize] => 1804
                    [mtime] => 1450276722
                    [filename_hash] => sVG21x9WNvrB-RJ68lpPFU1c1nC69Q6vpoE0ZOCXYhQ
                    [content_hash] => re2u0k4LnMflnE_vdtbf895Qi_SkkP2LsRyeZRrC4hQ
                    [linecount] => 52
                    [data] => modules/contextual/contextual.js
                    [fileext] => js
                    [cache_id] => advagg:file:sVG21x9WNvrB-RJ68lpPFU1c1nC69Q6vpoE0ZOCXYhQ
                )

            [sites/all/modules/contrib/jquery_update/replace/misc/jquery.form.min.js] => Array
                (
                    [filesize] => 10594
                    [mtime] => 1450276723
                    [filename_hash] => -2nlfPkds-PeNi-TwIp315clJ-HShrTMUWaGUzNHBDk
                    [content_hash] => zrdc5QneENbHKJYZrS_CG439iKiONm34SFfC5jKU-F8
                    [linecount] => 10
                    [data] => sites/all/modules/contrib/jquery_update/replace/misc/jquery.form.min.js
                    [fileext] => js
                    [cache_id] => advagg:file:-2nlfPkds-PeNi-TwIp315clJ-HShrTMUWaGUzNHBDk
                )

            [sites/all/modules/contrib/views/js/jquery.ui.dialog.patch.js] => Array
                (
                    [filesize] => 1182
                    [mtime] => 1450276723
                    [filename_hash] => 27VuufBZ1fX6jvvIoDMTkI1d-lHd7QsTh-tUhZE3xGw
                    [content_hash] => U1ynZDKcn0SaDsyNXmDwRnz57Bh-H_McW5X9KUfWnrg
                    [linecount] => 27
                    [data] => sites/all/modules/contrib/views/js/jquery.ui.dialog.patch.js
                    [fileext] => js
                    [cache_id] => advagg:file:27VuufBZ1fX6jvvIoDMTkI1d-lHd7QsTh-tUhZE3xGw
                )

            [sites/all/modules/contrib/token/token.js] => Array
                (
                    [filesize] => 2980
                    [mtime] => 1450276723
                    [filename_hash] => is_d3NE-y8GDPg3Zz4UAIXV2XfyIQQOdu8qdbTy0Vd0
                    [content_hash] => SWhgc68aUP4g9KhUWIE0T__sUSAPfzvbUBtSxFYvZHY
                    [linecount] => 87
                    [data] => sites/all/modules/contrib/token/token.js
                    [fileext] => js
                    [cache_id] => advagg:file:is_d3NE-y8GDPg3Zz4UAIXV2XfyIQQOdu8qdbTy0Vd0
                )

            [misc/form.js] => Array
                (
                    [filesize] => 2460
                    [mtime] => 1450276722
                    [filename_hash] => iNq7saNX8iaIXFQKkCLYfDxlmrn_oyC98jPLHTfPLjw
                    [content_hash] => uagYkBY5L2ttNjbunTXZasaLg3L2C9WjlxwNubeA3K4
                    [linecount] => 78
                    [data] => misc/form.js
                    [fileext] => js
                    [cache_id] => advagg:file:iNq7saNX8iaIXFQKkCLYfDxlmrn_oyC98jPLHTfPLjw
                )

            [misc/ajax.js] => Array
                (
                    [filesize] => 25017
                    [mtime] => 1450276722
                    [filename_hash] => iKnaOK61DIrdI--cleGK5s98K2cZWzAgC6-zYs0vpOA
                    [content_hash] => LqvhuMbf3pLq5HoX7phJWA_r1QpssnS0eczwY5fBVM0
                    [linecount] => 677
                    [data] => misc/ajax.js
                    [fileext] => js
                    [cache_id] => advagg:file:iKnaOK61DIrdI--cleGK5s98K2cZWzAgC6-zYs0vpOA
                )

            [sites/all/modules/contrib/jquery_update/js/jquery_update.js] => Array
                (
                    [filesize] => 302
                    [mtime] => 1450276723
                    [filename_hash] => gRfGHcNBOJmcbzFREEMytZcNUPe8UunKWfMbwyi9eGU
                    [content_hash] => -jhdxDgl_J9yMVOtCoRetm1vBOGgnHFpF4H3zzM6Su8
                    [linecount] => 7
                    [data] => sites/all/modules/contrib/jquery_update/js/jquery_update.js
                    [fileext] => js
                    [cache_id] => advagg:file:gRfGHcNBOJmcbzFREEMytZcNUPe8UunKWfMbwyi9eGU
                )

        )

)

@mr.baileys
Thanks for the detailed info. I have checks that should delete an aggregate file it its size ends up being zero inside of advagg_save_aggregate() and I have some other checks with rename and write failures inside of advagg_save_data(). The use of rename is done with temporary file names to ensure atomic file operations. What you're seeing tells me that somehow this is still happening even though I check for this. Where else can I add a check in the file writing process? It appears I'm missing something. The only other option I can think of is that the filesystem is lying to php/advagg saying write is good, filesize is not zero and then later on that not being true.

If the filesystem is lying then having a cron job remove zero byte files; having the aggressive cache check the disk (slow); and somehow schedule a filesize check 2 minutes after a write are the only ways I can see to work around a buggy filesystem. Open to ideas here.

You are using the latest version or latest dev correct?

Added more reporting in regards to file writes; advagg will now test a file write on the status report page and it should give some detailed info back if it fails #2645582: advagg creates lots of file_advagg js files in my sites/default/files folder

#21 was helpful but I couldn't say I found the smoking gun. Anyone having this issue and using the latest dev of AdvAgg?

I was able to reproduce part of this bug on our production environment. The new watchdog messages reported exactly what happened. If you have 3 or more webheads then this can affect you, depending on how you deploy code. This doesn't affect core because it doesn't use image style file generation.

Long story short a deployment happens; request hits the new code box and the aggregate request hits the old code box. Generally with 2 webheads during a deployment only one box will be active at a time so this can't happen; once you get to 3 or more webheads then you have the issue where old code and new code are running at the same time.

Thinking that turning off "Use HTTPRL to generate aggregates." will fix most of the occurrences of this issue but doesn't guarantee that it will not happen. Pushing out a new file is where this issue is likely to happen as the aggregate cache will not be valid and thus it will have to be created.

In terms of a fix I can have it not write the aggregate file if reading from the filesystem failed on a file not in the public directory.

This looks like the same issue I ran into with agrcache, see #2628918: Address deployment race conditions.

@catch
Thanks for pointing to that issue; it's re-assuring that we both came to the same conclusion in terms of not writing the file. Good idea on using the filename checksum as a final check before writing the file; might be a little tricky with core css compression so I'll be sure to look into this at a deep level.

Got the first version of this patch done. Will be fixing a couple of things; one of them is fixing SRI, got some bad SRI hashes on a second page load (first will be inevitable). Noted that after an hour of the file missing or of the hash not matching it will write the aggregate to disk, this is the advagg_file_read_failure_timeout setting that will need to be documented in the readme.

This should be fixed with this committed patch. Worse case with SRI is it will fail on the first load but work on the second.

Fixed a php notice