404 error on css/js aggregated files in site located in a VirtualDocumentRoot

I found a copy of D8 dev dated 4/27. It displayed the bartik themed front page as expected after install completed.

It looks like something in #2226761: Change all default settings and config to fast/safe production values caused it. Backing that out resulted in a correct display.

It would help if someone else can verify they can reproduce this problem. I've spent some time trying to figure this out and used a fresh git clone each time I tested something, so I don't think it is just me, but still...
I run on OS X 10.6 and normally do a git clone of D8 into my 'Sites' directory. I use SQlite and don't bother about 'clean' url's because it is convenient. I can build a new copy of D8, modify, test, save my work, and then flush and restart pretty quickly if I want.

#7 I was hoping but that's not it. The directories ../files/(css,js) both have 775 permissions and are full of files.
#6 Your file structure is not clear to me but isn't htdocs webroot?

You are right. The asset files all are using clean url's, eg:

<script src="http://localhost/~rocks/drupal8/core/assets/vendor/modernizr/modernizr.min.js?v=2.6.2"></script>
<script src="http://localhost/~rocks/drupal8/sites/default/files/js/js_rzFdFexjF52mPdXvnDvCi_2hzwlqa-qNXT2aurY12AY.js"></script>

I've attached 2 files withe header part of the markup. It appears clean url's are always used in the header.

These are the errors I see in the web console when I load the front page. Those files do exist.

GET http://localhost/~rocks/drupal8/index.php [HTTP/1.1 200 OK 515ms]
GET http://localhost/~rocks/drupal8/sites/default/files/css/css_loDEiBI2jYaEL_57KPk22T1TngQ_8Na-jMUI7TeleJU.css [HTTP/1.1 404 Not Found 3ms]
GET http://localhost/~rocks/drupal8/sites/default/files/css/css_-jKAAPSvcmNqtHc8neNxCJsYObaacSUbPBhA75TBI6A.css [HTTP/1.1 404 Not Found 2ms]
GET http://localhost/~rocks/drupal8/sites/default/files/css/css_H6QQIWUiqzix07EwaXzLoH15sjhkIPbLshcglFVgpJY.css [HTTP/1.1 404 Not Found 897ms]
GET http://localhost/~rocks/drupal8/sites/default/files/css/css_ThUn5DDTasQHIMv1MQAwyMj14lRkcVK5IsdlQeJkApY.css [HTTP/1.1 404 Not Found 1806ms]

When I log in I get errors on both the css and js files. Do not understand why those can't be found.

I see the files clearly, the permissions are correct. I can open the file in the browser as a file(file:///Users/rocks/Sites/drupal8/sites/default/files/css/..) but I can't open it as a url. The only other thing I can think of is if my web server is not configured properly, but that doesn't make sense since it works fine if I modify .htaccess to support clean url's. I tried entering the full url for the file directly in the browser and still not found. However I think I get drupal's 404 msg when I add 'index.php' to the url:

Not Found

The requested URL "http://localhost/~rocks/drupal8/index.php/sites/default/files/css/css_loDEiBI2jYaEL_57KPk22T1TngQ_8Na-jMUI7TeleJU.css?0" was not found on this server.

Is it possibly a .htaccess problem?

I don't usually modify either. However to get 'clean' url's on the mac I make this change to .htaccess

  # Modify the RewriteBase if you are using Drupal in a subdirectory or in a
  # VirtualDocumentRoot and the rewrite rules are not working properly.
  # For example if your site is at http://example.com/drupal uncomment and
  # modify the following line:
  # RewriteBase /drupal
  RewriteBase /~rocks/drupal8
  #

I've always assumed that OS X uses virtual document roots to achieve what they call 'Personal Web Servers' so that every user can create web sites in their local file system on a mac running their web server software. The ~(user name) seems to be the key for the web server to route url's to the right document root. I know no one has reproduced this yet, but If someone with a mac could try it might make me less crazy.

Actually, I am really running drupal in a sub directory. The real 'VirtualDocumentRoot' on a mac is '/Users/{user name}/Sites'. I usually have more than one web site in that directory that I am working on.

Based on previous comments #2226761: Change all default settings and config to fast/safe production values did not create but made apparent this problem since, by default, a new drupal install will now use aggregated css/js files. So I still have a problem with 'dirty' url's(unless another mac user succeeds where I have failed).

Re; #22 I had tried that before to no affect but did some more checking and the instructions are wrong. The settings.local.php file needs to be in ../sites/default, nit ../sites.
I will try D7 later today.

I installed the latest D7 dev, turned on aggregation, 'clean' url's off, but did not see the problem

I have a laptop running OS X 10.9.2, php 5.4, and the default Apache server that comes with OS X. I did a git clone of D8, using an identical setup as my desktop. And saw the same problem. My desktop runs OS X 10.6, php 5.4, and OS X Server, so there are some additional bells and whistles for the server environment. I don't think this is Mac specific but I don't have other systems yet to test.

Commenting out the following code from .htaccess seems to have solved the problem.

    # Serve gzip compressed CSS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

    # Serve gzip compressed JS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

The files were loaded properly without error. Testing was not extensive but the site loaded and displayed as expected.

ps. This code is exactly the same as in D7's .htaccess, which doesn't display the problem.

I know very little about rewrite rules and the .htaccess file but changing

    RewriteCond %{REQUEST_FILENAME}\.gz -s

to

    RewriteCond %{LA-U:REQUEST_FILENAME}\.gz -s

seems to work. Could someone who is more knowledgeable about this take a look at it?

Changing .htaccess as follows

    RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME}\.gz -s

also works. I'm not exactly sure what is going on here.

I have to take back what I said in #25. This fails on D7 as well, with the same 404 errors The same changes to .htaccess make it go away. Now I have to research D7 to see if this has been already reported.

There were several issues in D7 with similar symptoms. I did some more experimenting and think I have narrowed this down. I tested D8 on a redhat system I have access to and did not get the failures. But there d8 was installed as a sub directory of Document_Root, as were the tests in #6. So I tried installing d8 on Document_Root of my desktop, running OS X Server. It also didn't fail. So far this only fails for me if I use a Virtual_Document_Root, which is actually my normal development environment. I just never ran into this before because I never had css/js aggregation on before.
So the big question is whether running drupal in a Virtual_Document_Root environment happens in the real world.

Well, it gets worse. It only fails if you are running in a sub directory of a Virtual_Document_Root. Don't know what to do with this now. Multiple ways for me to get around it, so just an annoyance now. Is this going to be a real world problem for anyone?

That had no affect. The changes in #30 and #31 worked but I don't know the implications of doing those things. Clearly you don't change .htaccces lightly and I'm not knowledgeable enough on that. A clue may be that firefox's web console shows the 'GET' on the '404' request as file type 'html', while when I enable clean URL's the file type is 'css' on the successful 'GET's.

Sorry take the last back. All the rewrite rules following the 'RewriteCond %{REQUEST_FILENAME}\.gz -s' are not being executed.

No change. I thought the problem WAS that the files were not being recognized. The lines

    RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
    RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

aren't executing because I can see the content types aren't changing in firefox'sweb concole output.

I will say up front that I don't know what I'm doing here, but changing the existing rule, or the one you suggested, to add a $ so

    RewriteRule ^(.*)$\.css $1\.css\.gz [QSA]

works. Would a revision like

    # Serve gzip compressed CSS and JS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)$\.css$ $1.css.gz
    RewriteRule ^(.*)$\.js$ $1.js.gz

be acceptable or would it be better to minimize changes to the fewest bytes possible?

I probably should put the $ on other side. ...^(.*)\$.css.. works as well. I tried with both dirty and clean url's and everything appeared fine after some trivial testing. What do you think?

The .htaccess code with respect to aggregation is identical in D7 to D8. D7 also has no problem on my system when aggregation is turned on. So this is a regression from D7. It hasn't been confirmed yet that this doesn't happen on all macs nor the specific situation of a D8 site located in a virtual document root. I have confirmed that there is no problem when I install D8 in a sub directory of the real document root of the server running on my system.
D8 now assumes clean url's. And that seems to be where the problem lies,

Still can't get a handle on this, and it keeps getting more complicated. I just spent the last week plus installing a new mac pro for my desktop and finally got ~username sites working again. But the problem doesn't appear. So my test systems are:

1) A Macbook running OS X 10.9.3 with ~username sites enabled. Drupal 8 fails as above.
2) A Mac Pro with OS X 10.6.8 and Server 10.6.5 with ~username sites enabled. Drupal 8 fails as above. However Drupal 7 with cs and js aggregation enabled runs fine.
3) A new Mac Pro with OS X 10.9.3 and Server 3.1.2 with ~username sites enabled. Drupal 8 runs fine.

I don't think it is a .htacces problem as they are identical(re cs and js aggregation) in all cases. And I don't have any spurious .htaccess files laying around. However, OS X Server 3.1.2 is pretty complicated and finds configuration files in both its own directory and as well as the etc/apache2 directory. But I will take some time looking at this.

The machine running OS X 10.6 is running apache 2.2.24. The 2 systems running OS X 10.9 run apache 2.2.26. But the newest system uses a httpd.conf that is heavily customized for Apple. That is where I am looking now.

Well, I should have looked closer. When I installed my new system with OS X 10.9.3 it created a default ~username.conf file which I should have looked at more closely. It had set 'AllowOverride None'. So again, it appears the problem is the .htaccess file when it has to process a file that is off the 'VirtualDocumentRoot'. If I ignore the .htaccess file, the site has no problem with the aggregated css/js files. But, the .htaccess code specifically for css/js aggregation is identical for D7 and D8, yet D7 works and D8 doesn't . Where should I look.

FWIW

REQUEST_FILENAME
    The full local filesystem path to the file or script matching the request, if this has already been determined by the server at the time REQUEST_FILENAME is referenced. Otherwise, such as when used in virtual host context, the same value as REQUEST_URI. Depending on the value of AcceptPathInfo, the server may have only used some leading components of the REQUEST_URI to map the request to a file. 

This is the msg I see in Apache's error log: File does not exist: /Library/WebServer/Documents/Users , one for each of the aggregated files D8 is trying to load.

Everything points to the 'RewriteCond %{REQUEST_FILENAME}' test failing but I am not sure of the implications of this change in other environments. I am limited as to the environments I can test so I thought I would see what happens in the testbot. I also included the changes recommended in #39.

To me, the problem seems to be that the RewriteCond is what is failing. One little change, which at least makes sense to me, is all that is needed. If you change

   {old} RewriteCond %{REQUEST_FILENAME}\.gz -s

   {new} RewriteCond %{REQUEST_FILENAME}\.gz$ -s

Someone please tell me that it makes sense to them also.

OK, let's see if I understand what is going on. As per the quote from the apache manual in #48, when operating in a VirtualDocumentRoot {REQUEST_FILENAME} returns the same info as {REQUEST_URI}. In my case it is returning the /Library/WebServer/Documents/Users string I see in my apache error log, one for each of the cs/js aggregated files drupal is trying to load. This implies that once the RewriteCond %{REQUEST_FILENAME}\.gz -s statement is executed the above string is now the {REQUEST_FILENAME} object that is returned from .htaccess, resulting in the 404 error.

Some more quotes from the apache manual:

The variables SCRIPT_FILENAME and REQUEST_FILENAME contain the same value - the value of the filename field of the internal request_rec structure of the Apache HTTP Server. The first name is the commonly known CGI variable name while the second is the appropriate counterpart of REQUEST_URI (which contains the value of the uri field of request_rec).

If a substitution occurred and the rewriting continues, the value of both variables will be updated accordingly.

If used in per-server context (i.e., before the request is mapped to the filesystem) SCRIPT_FILENAME and REQUEST_FILENAME cannot contain the full local filesystem path since the path is unknown at this stage of processing. Both variables will initially contain the value of REQUEST_URI in that case. In order to obtain the full local filesystem path of the request in per-server context, use an URL-based look-ahead %{LA-U:REQUEST_FILENAME} to determine the final value of REQUEST_FILENAME.

I think the middle statement is the most important here and explains what is happening in my case. It also suggests the solution.

Clarify issue.

This fixes the original issues, but doesn't address any of the other problems pointed out in the .htaccess file in preceding comments. It might also be appropriate to look at other code in the current .htaccess file using the {REQUEST_FILENAME} variable.

Sorry, I know it is the same patch, but I learned a lot in between.

This still happens. This patch is a little more general and I hope it satisfies all needs.

Well, this doesn't work either. It looks like css/js aggregation simply won't work without clean url's in a VirtualDocumentRoot.

I did check the patch in #50 at simplytest.me and it worked there, though that is not testing a VirtualDocumentRoot environment, so at least nothing is broken. Has anybody actually verified that the code in .htaccess to support css/gz aggregation actually works? When I use firefox's web console I don't see any .gz files being loaded and the sizes of the css files being loaded correspond to the unzipped versions of the file. It's the same on simplytest.me as well as my system.

Testing this as best as I could with firefox's web console, the only thing I can say for sure is that this doesn't happen in a 'clean urls' environment. I can achieve this by modifying .htaccess to activate "RewriteBase /mylocalpath". I don't think any other changes to .htaccess are necessary, nor have any others that I have tried truly worked. But this is still a regression from D7. D8 assumes clean urls. Is that going to become required?

I will keep looking though.

It is really simple. I do a git clone of 8.0.0 in my 'Sites' directory and go thru install. On a Mac this means I am running in a VirtualDocumentRoot and WITHOUT clean urls. This problem only occurs when you are NOT running clean urls. Since you did specify a RewriteBase, that means on your system D8 is running in a VirtualDocumentRoot but WITH clean urls. On a mac mod_alias is enabled by default. In fact, my server would not serve up anything, even localhost, with mod_alias disabled, probably because of some other config statements. Since D7 allowed drupal to run in a VirtualDocumentRoot and WITHOUT clean urls, I'm going to take a look at D7 to see if I can figure out what is different.

Mod_rewrite is default on the mac. You could use VirtualDocumentRoot without RewriteBase in Drupal 7. You did have to add index.php to your urls or you couldn't find any pages except the home page. In D8, this worked until css/js aggregation = On became the default setting. As you said, in a VirtualDocumentRoot, clean urls = mod_rewrite + RewriteBase. This is a regression because in D7 I could have css/js aggregation with un-clean urls while in d8 I can't.

Perhaps I didn't push D7 enough, but as long as I didn't mind the ugly url, it always worked fine for me. I know that clean urls are assumed now in D8, though I haven't really seen that documented anywhere. It isn't even a configuration option anymore. But if it is required now, then that does need to be documented somewhere and install should test for it. If it is required, then I don't have an issue. If it is not, then this is a regression.

How many inexperienced drupal users are going to pick that up, and where does it say MUST. These are the same lines as in the drupal 7 .htaccess. I've read these ambiguous lines many times, they do not mention clean urls, and, in D7, the (unspecified) rewrite rules did seem to work. If this is truly a statement of a requirement it needs to be someplace else(README.txt?), written more forcefully, and install should test for this.

I know you have more important things to work on so I am sorry to let this drag on. The only place this is fatal is if you build a site somewhere you don't have access to .htaccess. Maybe all this needs is more clarity.

It's not up to me to assert that clean urls are required in drupal 8. But judging by the behavior that seems to be the case. There is still a lot of code in D8 that that tests whether clean urls are active or not. But css/js aggregation does not work unless they are in a clean url environment. Whether there are other things that don't work I don't know.

I should think of change records sooner than I have, since I did find it 7 months ago. But to me it implies that un-clean urls still have some support. Then this problem with css/js aggregation is maybe a new followup like #1546082: Follow-up to variable_get('clean_url') removal, as it seems quite unlike the problems discussed in the issues you have pointed out.

I have seen it maybe twice in the last year but currently am unable to repeat it. Others have posted issues with similar symptoms but looked unrelated to this. Unfortunately I've lost track of those. I still have problems with dirty URL's but I haven't spent time on that because I just do workarounds(see #2385617: Dirty (index.php/ based) Urls are not consistently generated

Even a corrected .htaccess wasn't working for a while, which is what I had been doing for a long time. That is why I opened issue. Somewhere something changed while this issue was open and then everything was as before. So be it.