Add warning to detect if local cache not in sync with S3 file system

I should also add that since the libraries files are not located in advagg_files, they do not show up in the "Per Files" options for JS compression.

After further investigation, the issue is a bit more specific.

For some reason, the Drupal system cache is not in sync with the current state of all S3 files. There are cases where a single advagg_js file has already been removed from S3 (does not exist at all), yet a Drupal page is still trying to load it.

Have also seen this in the Status Report where it will intermittently say that gzip is not working for JS files. However on closer examination, the file no longer exists in S3, and Drupal is trying to load it.

Have tried clearing the system cache, the AdvAgg cache, AdvAgg files, and running cron multiple times, but it still attempts to call an old file.

Just loaded the Status Report page and here is an example... getting an "Access Denied" on a file that doesn't exist in S3.

The web server is not returning a 200, instead a 403 is being returned. gzip can not be tested. Raw request info:

stdClass Object
(
    [url] => https://localhost/s3fs-js/advagg_js/js__ChUT0Cpt3ch0T35nhZgMgH5BzBQ5oyaFF27cnXxe7aU__AXOPhchCc8rks5Oj8q6ca6kvFPfZvWueAugL8Ai5FfU__SjbHX4dzJ01Nm_Z_NW3iHrxdlH-8AlaJOg08gTcT4JI.js
    [status] => Done.
    [code] => 403
    [chunk_size] => 32768
    [data] => f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>FA64802B6C68AA55</RequestId><HostId>/AqJdWt3i1uXB1hwGI/VW2rsPn+LedRGomcHtMagTuytw5IKZh6adRB/8NHgzx5hpUY1348rnKU=</HostId></Error>
0
    [request] => GET /s3fs-js/advagg_js/js__ChUT0Cpt3ch0T35nhZgMgH5BzBQ5oyaFF27cnXxe7aU__AXOPhchCc8rks5Oj8q6ca6kvFPfZvWueAugL8Ai5FfU__SjbHX4dzJ01Nm_Z_NW3iHrxdlH-8AlaJOg08gTcT4JI.js HTTP/1.1
Accept-Encoding: gzip, deflate
Connection: close
Referer: https://localhost/admin/reports/status
User-Agent: Drupal (+http://drupal.org/)
Host: localhost

....

With some additional testing, I'm able to narrow this down further.

If "Use HTTPRL to generate aggregates" is enabled, the files are constantly out-of-sync between the AdvAgg database tables and what is stored in S3.

After disabling, the Status Report has no issues, and do not receive error messages on other pages due to missing files.

Cross-referencing to this issue because it's very similar: https://www.drupal.org/project/advagg/issues/2593243

In comment #2 it was suggested to set $conf['advagg_skip_gzip_check'] = TRUE;. I don't think that is a good idea since it just masks the issue.

I'm wondering if HTTPRL creates the aggregates too quickly, and some of them don't successfully get delivered to S3?

This is still occasionally an issue with "Use HTTPRL to generate aggregates" disabled, but not nearly as bad as when enabled.

Also I'm not sure why, but when AdvAgg is enabled for node create/edit pages with S3, it is completely locking up the server and eventually we get an Nginx timeout.

Whatever is happening during that time, the advagg_aggregates table now has over 20,000 (!) records. Didn't have any of these problems when using the local file system.

The attached patch does not solve the root issue, but provides a way to debug for the problem in the Status Report. Maybe this can be of help to others.

Before performing any drupal_http_request, a check is done to see if the file actually exists in the S3 file system.

After applying this patch, I never see the gzip warning message. It is always replaced with a "file does not exist" warning.

I've added patches for the two items that I believe are causing the local DB to be out-of-sync with S3 files:

#2973383: Use public rename function included with s3fs: https://www.drupal.org/project/advagg/issues/2973383
#2973392: Use file_scan_directory to find first advagg file for s3fs: https://www.drupal.org/project/advagg/issues/2973392

Since adding the patches, have not seen any further problems with this issue, and getting a clean Status Report.

Leaving this patch as a separate item to review, since I think it still might be worthwhile to have this warning message in place if something else comes up which causes the two to be out-of-sync.

Still having HTTPRL issues on node create/edit pages, but going to spin that off into a separate issue.

This is definitely a HTTPRL + AdvAgg + HTTPS issue. I'm going to address further in Issue #2973446: https://www.drupal.org/project/advagg/issues/2973446

I've modified this patch to add extra info if a site is running HTTPRL and using it to aggregate AdvAgg files.