In Drupal status page, I got these messages:

Adv CSS/JS Agg - HTTP Request	HTTP requests to advagg for css files are not getting through.
AdvAgg will issue a request for a file that does not exist inside of the AdvAgg directory. If AdvAgg sends a 404, everything is ok; if something else sends a 404 then that means that AdvAgg will not be able to generate an aggregate if it is missing as something else is handling the 404 before AdvAgg has a chance to do it. If you are reading this, it means that something else is handling the 404 before AdvAgg can. Raw request info:


Adv CSS/JS Agg - HTTP Request	HTTP requests to advagg for js files are not getting through.
AdvAgg will issue a request for a file that does not exist inside of the AdvAgg directory. If AdvAgg sends a 404, everything is ok; if something else sends a 404 then that means that AdvAgg will not be able to generate an aggregate if it is missing as something else is handling the 404 before AdvAgg has a chance to do it. If you are reading this, it means that something else is handling the 404 before AdvAgg can. Raw request info: 

There is no "Raw request info", I do not know how should I debug it...

Website is hosted on shared Apache hosting, PHP 5.5.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Talkless created an issue. See original summary.

mikeytown2’s picture

Status: Active » Needs review
FileSize
1.49 KB
DamienMcKenna’s picture

So it's a difference between print_r() and var_export()?

mikeytown2’s picture

print_r doesn't tell me if its FALSE, NULL or an empty string. I'm looking over the code and I can't figure out how drupal_http_request would return nothing; at a minimum an object should have been returned.

mikeytown2’s picture

Version: 7.x-2.14 » 7.x-2.x-dev
Category: Bug report » Support request
Status: Needs review » Active

Patch was committed. Something should be there now in the latest dev.

Talkless’s picture

I have just tried dev version, info is printed now, thank you!

Now, I have to figure out why apache handles "not found", and not advagg/drupal..? But that's another story.

mikeytown2’s picture

Quote from here: #2062547-11: Better Explain "HTTP requests to advagg for css/js files are not getting through"

stdClass::__set_state(array(
   'request' => 'GET /sites/default/files/advagg_js/js__1440693838.js HTTP/1.0
Connection: close
Referer: http://beta.MYSITE.com/admin/reports/status
User-Agent: Drupal (+http://drupal.org/)
Host: 192.168.0.200

',
   'data' => '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /sites/default/files/advagg_js/js__1440693838.js was not found on this server.</p>
<hr>
<address>Apache/2.2.22 (Ubuntu) Server at 192.168.0.200 Port 80</address>
</body></html>
',
   'protocol' => 'HTTP/1.1',
   'status_message' => 'Not Found',
   'headers' => 
  array (
    'date' => 'Thu, 27 Aug 2015 16:44:07 GMT',
    'server' => 'Apache/2.2.22 (Ubuntu)',
    'vary' => 'Accept-Encoding',
    'content-length' => '324',
    'connection' => 'close',
    'content-type' => 'text/html; charset=iso-8859-1',
  ),
   'code' => '404',
   'error' => 'Not Found',
))

Same for CSS

Do you use image styles?

Talkless’s picture

Yes, I use Image Styles, Picture module with picture mappings. But, how does this relate?

mikeytown2’s picture

image styles are located in the
sites/default/files/styles/dir;
advagg is located in the
sites/default/files/advagg_css/ and
sites/default/files/advagg_js/ dir.
So that means that your web server configuration allows for non existing files to be created in the styles dir.

Something else I just thought about would be the ability to put the advagg dirs inside the styles dir. A little bit of a hack but would be an alternative to the current way of asking to look that the htaccess file on a per case basis.

I would make the code below an option or better yet have a dir variable that can be set, default is blank.

/**
 * Implements hook_advagg_get_root_files_dir_alter().
 */
function advagg_advagg_get_root_files_dir_alter(&$css_paths, &$js_paths) {
  $css_paths[0] = 'public://styles/advagg_css';
  $js_paths[0] = 'public://styles/advagg_js';

  file_prepare_directory($css_paths[0], FILE_CREATE_DIRECTORY);
  file_prepare_directory($js_paths[0], FILE_CREATE_DIRECTORY);

  // Set the URI of the directory.
  $css_paths[1] = advagg_get_relative_path($css_paths[0]);
  $js_paths[1] = advagg_get_relative_path($js_paths[0]);
}

What's your htaccess file look like?

mikeytown2’s picture

Status: Active » Needs review
FileSize
1.94 KB

With this patch add this to your settings.php file

$conf['advagg_root_dir_prefix'] = 'public://styles/';

Fixing your htaccess file would be ideal, so posting it here for analysis would be the better option.

Talkless’s picture

Here's htaccess. Only thing changed should be redirecting "www.something.com" to "something.com", and at the end I have appended advagg some sort of FileMatch rules from README.txt:

#
# Apache/PHP/Drupal settings:
#

# Protect files and directories from prying eyes.
<FilesMatch "\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\..*|Entries.*|Repository|Root|Tag|Template)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig\.save)$">
  Order allow,deny
</FilesMatch>

# Don't show directory listings for URLs which map to a directory.
Options -Indexes

# Follow symbolic links in this directory.
Options +FollowSymLinks

# Make Drupal handle any 404 errors.
ErrorDocument 404 /index.php

# Set the default handler.
DirectoryIndex index.php index.html index.htm

# Override PHP settings that cannot be changed at runtime. See
# sites/default/default.settings.php and drupal_environment_initialize() in
# includes/bootstrap.inc for settings that can be changed at runtime.

# PHP 5, Apache 1 and 2.
<IfModule mod_php5.c>
  php_flag magic_quotes_gpc                 off
  php_flag magic_quotes_sybase              off
  php_flag register_globals                 off
  php_flag session.auto_start               off
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_flag mbstring.encoding_translation    off
</IfModule>

# Requires mod_expires to be enabled.
<IfModule mod_expires.c>
  # Enable expirations.
  ExpiresActive On

  # Cache all files for 2 weeks after access (A).
  ExpiresDefault A1209600

  <FilesMatch \.php$>
    # Do not allow PHP scripts to be cached unless they explicitly send cache
    # headers themselves. Otherwise all scripts would have to overwrite the
    # headers set by mod_expires if they want another caching behavior. This may
    # fail if an error occurs early in the bootstrap process, and it may cause
    # problems if a non-Drupal PHP file is installed in a subdirectory.
    ExpiresActive Off
  </FilesMatch>
</IfModule>

# Various rewrite rules.
<IfModule mod_rewrite.c>
  RewriteEngine on

  # Set "protossl" to "s" if we were accessed via https://.  This is used later
  # if you enable "www." stripping or enforcement, in order to ensure that
  # you don't bounce between http and https.
  RewriteRule ^ - [E=protossl]
  RewriteCond %{HTTPS} on
  RewriteRule ^ - [E=protossl:s]

  # Make sure Authorization HTTP header is available to PHP
  # even when running as CGI or FastCGI.
  RewriteRule ^ - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

  # Block access to "hidden" directories whose names begin with a period. This
  # includes directories used by version control systems such as Subversion or
  # Git to store control files. Files whose names begin with a period, as well
  # as the control files used by CVS, are protected by the FilesMatch directive
  # above.
  #
  # NOTE: This only works when mod_rewrite is loaded. Without mod_rewrite, it is
  # not possible to block access to entire directories from .htaccess, because
  # <DirectoryMatch> is not allowed here.
  #
  # If you do not have mod_rewrite installed, you should remove these
  # directories from your webroot or otherwise protect them from being
  # downloaded.
  RewriteRule "(^|/)\." - [F]

  # If your site can be accessed both with and without the 'www.' prefix, you
  # can use one of the following settings to redirect users to your preferred
  # URL, either WITH or WITHOUT the 'www.' prefix. Choose ONLY one option:
  #
  # To redirect all users to access the site WITH the 'www.' prefix,
  # (http://example.com/... will be redirected to http://www.example.com/...)
  # uncomment the following:
  # RewriteCond %{HTTP_HOST} .
  # RewriteCond %{HTTP_HOST} !^www\. [NC]
  # RewriteRule ^ http%{ENV:protossl}://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
  #
  # To redirect all users to access the site WITHOUT the 'www.' prefix,
  # (http://www.example.com/... will be redirected to http://example.com/...)
  # uncomment the following:
  RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
  RewriteRule ^ http%{ENV:protossl}://%1%{REQUEST_URI} [L,R=301]

  # Modify the RewriteBase if you are using Drupal in a subdirectory or in a
  # VirtualDocumentRoot and the rewrite rules are not working properly.
  # For example if your site is at http://example.com/drupal uncomment and
  # modify the following line:
  # RewriteBase /drupal
  #
  # If your site is running in a VirtualDocumentRoot at http://example.com/,
  # uncomment the following line:
  # RewriteBase /

  # Pass all requests not referring directly to files in the filesystem to
  # index.php. Clean URLs are handled in drupal_environment_initialize().
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !=/favicon.ico
  RewriteRule ^ index.php [L]

  # Rules to correctly serve gzip compressed CSS and JS files.
  # Requires both mod_rewrite and mod_headers to be enabled.
  <IfModule mod_headers.c>
    # Serve gzip compressed CSS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

    # Serve gzip compressed JS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

    # Serve correct content types, and prevent mod_deflate double gzip.
    RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
    RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

    <FilesMatch "(\.js\.gz|\.css\.gz)$">
      # Serve correct encoding type.
      Header set Content-Encoding gzip
      # Force proxies to cache gzipped & non-gzipped css/js files separately.
      Header append Vary Accept-Encoding
    </FilesMatch>
  </IfModule>
</IfModule>






    <FilesMatch "^(css|js)__[A-Za-z0-9-_]{43}__[A-Za-z0-9-_]{43}__[A-Za-z0-9-_]{43}.(css|js)(\.gz)?">
      # No mod_headers
      <IfModule !mod_headers.c>
        # No mod_expires
        <IfModule !mod_expires.c>
          # Use ETags.
          FileETag MTime Size
        </IfModule>
      </IfModule>

      # Use Expires Directive.
      <IfModule mod_expires.c>
        # Do not use ETags.
        FileETag None
        # Enable expirations.
        ExpiresActive On
        # Cache all aggregated css/js files for 52 weeks after access (A).
        ExpiresDefault A31449600
      </IfModule>

      <IfModule mod_headers.c>
        # Do not use etags for cache validation.
        Header unset ETag
        <IfModule !mod_expires.c>
          # Set a far future Cache-Control header to 52 weeks.
          Header set Cache-Control "max-age=31449600, no-transform, public"
        </IfModule>
        <IfModule mod_expires.c>
          Header append Cache-Control "no-transform, public"
        </IfModule>
      </IfModule>
    </FilesMatch>
    # Force advagg .js file to have the type of application/javascript.
    <FilesMatch "^js__[A-Za-z0-9-_]{43}__[A-Za-z0-9-_]{43}__[A-Za-z0-9-_]{43}.js(\.gz)?">
      ForceType application/javascript
    </FilesMatch>
Talkless’s picture

I have applied patch and settings.php changes, but still got notfound handled by apache, though path has in fact changed:

<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /sites/default/files/styles/advagg_js/js__1440839912.js was not found on this server.</p>
<hr>
<address>Apache/2.2.22 (Ubuntu) Server at 192.168.0.200 Port 80</address>
</body></html>
mikeytown2’s picture

Status: Needs review » Active

hmmm that makes me wonder how on demand image generation works on your server if the styles dir is blocked. How in control are you of your web server?

Talkless’s picture

Currently I'm playing in my virtual machine of that I have full control, though originally site is published in shared php hosting server, where I can only upload files, create subdomain and change PHP version. No fiddling with Apache configuration and such, no shell.

Now, I did some experiments:

I deleted and recreated ...../files/styles directory. Cleared site cache, reloaded page, and all images, advagg_css and advagg_js files where generated!

Then I selected some specific css file, such as this:
/sites/default/files/styles/advagg_css/css__2w7im-0vtrklLpIAPbChuN5iE4Tnu-HLNh7-260GjP8__dqEsG3AI5OFGzLO0A4QyK21d56xaqawZV62STzG9cbc__mEt5Xw4eOZJ2SgZ9wyYfLh8ee26wLZtDWQwHM-E_k_c.css?redirect_counter=1

Deleted that CSS file in styles/advagg_css directory (actually, just deleted all CSS files there), and after entering that URL into browser, it got generated! (with .gz version also).

But, if I request for /sites/default/files/styles/advagg_css/css__1441212894.css file, it's not being generated, returning that "not found" Apache page.

Maybe it's something wrong with that test-css file generation?

Talkless’s picture

Some more info:

If I disable File Cache (filecache) module, or more precisely, if I comment-out these settings.php lines:

$conf['cache_backends'] = array('sites/all/modules/filecache/filecache.inc');
$conf['cache_default_class'] = 'DrupalFileCache';

I get these 404 errors in status page only once after clearing cache. If I reload /admin/reports/status later, there are no ore errors.

Then again, if I make File Cache module fully working, 404 errors reappear on every /admin/reports/status reload.

Also, here's line during status page reload form Apache error log:

[Tue Sep 08 20:50:35 2015] [error] [client 192.168.0.200] File does not exist: /var/www/sites, referer: http://beta.MYSITE.com/admin/reports/status

Path /var/www/sites is invalid, it should be at least /var/www/beta.MYSITE.com/sites

mikeytown2’s picture

What does echo filecache_directory(); give you?

The test file is there to make sure that drupal handles all requests to the /sites/default/files/styles/advagg_css/ dir; I use the current time as part of the filename.

[Tue Sep 08 20:50:35 2015] [error] [client 192.168.0.200] 
File does not exist: /var/www/sites, 
referer: http://beta.MYSITE.com/admin/reports/status

Thanks for providing this info. I think the issue is the loopback request is not setting the host correctly.

What are your the server values? (feel free to scrub most of this; I'm looking for hostnames and ip's.

var_export($_SERVER);
Talkless’s picture

filecache_directory() gives /var/www/beta.MYSITE.com/sites/default/files/.ht.filecachearray

$_SERVDER:

( 'protossl' => '', 'HTTP_AUTHORIZATION' => '', 'HTTP_HOST' => 'beta.MYSITE.com', 'HTTP_USER_AGENT' => 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0', 'HTTP_ACCEPT' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_LANGUAGE' => 'lt,en-US;q=0.8,en;q=0.6,ru;q=0.4,pl;q=0.2', 'HTTP_ACCEPT_ENCODING' => 'gzip, deflate', 'HTTP_DNT' => '1', 'HTTP_COOKIE' => '_pk_id.1.fa29=ef8e4f62d08326ac.1430466171.2.1432919423.1432918985.; Drupal.tableDrag.showWeight=0; SESS33cb4e43905a84df5da4697291a3fd14=eZV5yTHijbcx6hlVECdP2NYq1fF8xkCeUbWSWuDL_Mc; DrupalModuleFilter=activeTab%3Dall', 'HTTP_CONNECTION' => 'keep-alive', 'PATH' => '/usr/local/bin:/usr/bin:/bin', 'SERVER_SIGNATURE' => '
Apache/2.2.22 (Ubuntu) Server at beta.MYSITE.com Port 80
', 'SERVER_SOFTWARE' => 'Apache/2.2.22 (Ubuntu)', 'SERVER_NAME' => 'beta.MYSITE.com', 'SERVER_ADDR' => '192.168.0.200', 'SERVER_PORT' => '80', 'REMOTE_ADDR' => '192.168.0.10', 'DOCUMENT_ROOT' => '/var/www/beta.MYSITE.com', 'SERVER_ADMIN' => 'webmaster@localhost', 'SCRIPT_FILENAME' => '/var/www/beta.MYSITE.com/index.php', 'REMOTE_PORT' => '37757', 'GATEWAY_INTERFACE' => 'CGI/1.1', 'SERVER_PROTOCOL' => 'HTTP/1.1', 'REQUEST_METHOD' => 'GET', 'QUERY_STRING' => '', 'REQUEST_URI' => '/', 'SCRIPT_NAME' => '/index.php', 'PHP_SELF' => '/index.php', 'REQUEST_TIME' => 1441814460, 'HTTP_REFERER' => '', ) 

  • mikeytown2 committed a22b430 on 7.x-2.x
    Issue #2547367 by mikeytown2: host is set incorrectly on the status...
mikeytown2’s picture

Status: Active » Fixed
FileSize
1.27 KB

I think I figured this out. This patch has been committed. Let me know if this doesn't fix it.

mikeytown2’s picture

Also committed the patch in #10

Talkless’s picture

Now, with patch #19 I get different error in status page:

Adv CSS/JS Agg - Self Request Failure	

HTTP loopback requests to this server are returning a non positive response code of -1


If you have manually verified that AdvAgg is working correctly you can set the advagg_skip_404_check variable to TRUE in your settings.php. Editing the servers hosts file so the host name points to the localhost might also fix this (127.0.0.1 beta.MYSITE.com). To manually check go to http://beta.MYSITE.com/sites/default/files/styles/advagg_css/css__1442060272.css, view the source and check for this string <!-- advagg_missing_fast404 -->. If that string is in the source, you can safely add this to your settings.php file $conf['advagg_skip_404_check'] = TRUE; Enabling the HTTP Parallel Request and Threading Library module might be able to fix this as AdvAgg will use HTTPRL to build the URL if it is enabled.

No matter if Filecache is enabled or not. I do not see apache errors, only access line:
27.0.0.1 - - [12/Sep/2015:15:18:01 +0300] "GET /sites/default/files/styles/advagg_css/css__1442060272.css HTTP/1.0" 404 810 "http://beta.MYSITE.com/admin/reports/status" "Drupal (+http://drupal.org/)

I do, in fact, see "advagg_missing_fast404" inside "not found" page for css__1442060272.css.

BUT, after I deleted all advagg_css/* files, this error only occurs on first status page load after cache is cleared.

Could it be something related to the fact, that my site uses SQLite as it's database engine? Maybe it can't perform queries concurrently when performing these status page tests or something...?

EDIT: Looks like I get this error when I have installed "adminimal_admin_menu" and I can click "Flush caches" menu item while I'm in status page. Flush caches, status page reloads with "Adv CSS/JS Agg - Self Request Failure" error, but on next manual refresh status page do not show any previous errors.

mikeytown2’s picture

Status: Fixed » Active

Thanks for the feedback. Will look into this more.

mikeytown2’s picture

According to the documentation the error code you're getting is due to a local firewall rule.

Looked into the error of -1 which is actually 1 from stream_socket_client(); when a network error occurs, we use a negative number so it does not clash with the HTTP status codes, see drupal_http_request(). So stream_socket_client is returning a 1 which comes from the connect system call. By doing some looking around an error code of 1 usually means it can't run due to permission issues
http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/Err...
In this case the error means "The user tried to connect to a broadcast address without having the socket broadcast flag enabled or the connection request failed because of a local firewall rule." Pretty sure your server is not a Broadcast IP address so the firewall error is whats left.

Talkless’s picture

Maybe it's sort of Virtual machine issue. I'll try your latest patch on production website, maybe it will not have this strange issue.

Generally, I thinks this bug is fixed, I just would like to try it on shared hosting site what does not belong to me.

mikeytown2’s picture

Status: Active » Fixed

I'll mark this as fixed. Looking up the error code was something I needed to do to make sure there wasn't anything else that could be done in the advagg code.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.