default files directory not read by search engines

There was a lot of discussion over where to put the default files directory over at http://drupal.org/node/194369. This particular point did not come up that I remember.

Based on my understanding of the priority levels, I don't think this is critical as it does not break Drupal. Someone else can change the priority back if they disagree.

AFAIK, Allow is a Google extension. So I agree that's not a solution. Based on the discussion noted for the issue above, the current location of sites/default/files was the best option anyone came up with. I don't have a better idea for a solution right now, but I'll think on it.

There were plenty of reasons to go with sites/default/files. Original issue is here: http://drupal.org/node/191310

Changing the directory to something else would require at least one additional step in the installer - since they'd have to grant write permissions to their root drupal directory for it to write directly to /files or make the files directory manually and grant permissions to that.

I'd support documenting this in INSTALL.txt, and possibly a change to robots.txt if we don't mind an invalid one, since Google and Yahoo at a minimum respect the "Allow" directive. But I don't agree this is critical.

Cross-posted, back to normal.

Since Allow is not part of the robots.txt standard, I would rather not use it in the robots.txt supplied with the installation.

We still need to document this and have the person setting up the site make changes to support all of the other search engines. It might confuse them to say that Google and Yahoo will work, but go make changes so all the others will too.

+1 for updating INSTALL.txt with the appropriate information on this.

OK here's a first run.

mfer, it was a critical usability issue that led to the directory being created in sites/default/files in the first place. The current open issue looking at alternatives is here: http://drupal.org/node/98824 (also long).

Also, I should mention this location is specified in the RC2 announcement as of last night, so no way we can change it for D6.

If Disallow: /sites/default is only to save bandwidth, would it be too serious to remove the restriction? It's not a security issue, and the main reason for excluding it appears to be to avoid search engines unnecessarily crawling that directory if there's nothing worth looking at in there, that might be invalidated. We could still disallow sites/default/modules and sites/default/themes explicitely.

You mean sites/all/modules and sites/all/themes, right?

I meant both actually - I was under the impression some people use sites/default/themes (or modules) as well.

So:

Disallow: /sites/default/themes/
Disallow: /sites/default/modules/
Disallow: /sites/all/modules/
Disallow: /sites/all/themes/

??

If someone's setting up multisite, then they can sort out their own robots.txt file (or use the robots module).

With multisite you can still set your file directories to something like files/sitekey - it's just not documented very much. At the moment, I'd go with 2. Marking as needs work for those (sensible, IMO) changes to robots.txt

#10 is the alternative that I had thought of suggesting, mfer beat me to the punch.

-1 to #11, my thoughts on this are in #4. Also, I feel this discrepancy would generate numerous tickets. If someone's going to notice search engines not seeing their download directory, they'll be just as upset if it works partially.

If we went with the patch in #5, we'd need an accompanying documentation page to refer to with more information, or make updates to a possibly existing page if there's something appropriate already. I don't have the time to go look right now.

Or I could just roll a patch. Also took out INSTALL.txt changes since they wouldn't be valid with the robots changes.

Why do we block the sites folder at all in the first place?

robots.txt says:

# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.

I guess it's to prevent 403 errors and hits on theme pngs and js files? maybe?

This seems to be the original issue, where /sites just sorta' appeared in toto in a list with the other directories off of a root Drupal install.

Well I can't think of a good reason for it to be in there, so here's one to take it out. If someone really wants to restrict it from crawlers, there's nothing stopping them.

Since the standard is to place contrib modules and themes under the sites directory structure, why would we want to just open up the entire thing?

mfer's option to disallow the default folders included with core in #10 seems a better solution. It keeps the crawlers out of the recommended directories, but lets the files be picked up. This solution would require documenting the additional edits for multisites.

Alternative patch attached.

@ScoutBaker: I think the .htaccess directives protect most of the stuff in sites/ already (with the exception of images and such).

@keith.smith: Aren't there hosting sites that have restrictions on .htaccess? In that case, /sites would lose even that protection. Even where it works, I don't think having the entries in robots.txt hurts.

If we're going to remove robots.txt coverage for contrib modules and themes, then why are we protecting core using robots.txt? I don't know what the original reasoning to add the core directories was, but it seems likely that, if the reason is still valid, it would apply to contrib directories as well.

(Edit: cross-posted with mfer)

Granted. But if your .htaccess file doesn't working to protect those files (and clearly it does not on some systems), then wasn't your only measure of protection from this stuff being indexed previously whether or not a bot obeyed the robots.txt file? Or is there a piece of the puzzle I'm missing here?

(Edit: And I cross-posted with ScoutBaker)

Thing all .inc .php etc. files are going to be 403 for crawlers anyway. afaik this is to avoid unecessary 403 errors rather than security (and possibly to avoid some traffic on js and images).

Not everywhere, as said above. I'm on a hosting, where the .htaccess protection doesn't work (if I uncomment that in .htaccess, I get WSOD with internal server error). If I navigate (as anonymous user) to, say, includes/theme.inc on my site, I get the code as plain text to my screen. Nothing stops robots from indexing, once they get a link (not likely, but anyone can post it anywhere on the web). robots.txt is a chance to stop the robot, not for sure, but likely.

Ok I didn't know that :)

In that case #19 looks like a good option to me.

Attached is a patch with an initial attempt at documenting what I understand to be the behavior in #19. Note that this is likely to be pretty fragile, in that we're going to be depending on people to add these entries.

And, though large multi-site configurations likely do not need "one more thing" to have to worry about, I suspect most of them can or will automate this. Its the people like me who just do these every now and again that will almost certainly forget.

On the other hand, I'm not certain the world comes to an end if they (or I) don't do it.

+1 for #27. The additional text describing the updates is just what we needed.

Bleh. Even though it looks better the other way, this patch eliminates the extra leading space I had between the "#" and the directives in the commented-out example guidelines in robots.txt

For users whose .htaccess file works as intended could this be a bit "belt-and-braces"? For these users, this is surely only necessary if they don't want theme files (and some module files, e.g. CSS and images) crawled. On the other hand, is it possible some users might actually want e.g. theme images to be crawled? Maybe it would be helpful to qualify this statement a bit: "As you add sites to your configuration, you should edit the robots.txt file to include your site-specific configuration files and any module and theme directories." to explain the ins and outs...

It's also advertising the existence of the settings.php file, which in ?most cases is in anyway invisible/inaccessible. Is this wise? If it can be accessed directly then you've probably got a security problem anyway.

Sorry, my understanding of all the issues is too limited to do more than throw in these half thought-through comments!

gpk's right about the settings.php file, no way that should be listed.

This issue still applies to Drupal 7. The patch in #29 needs a reroll and to address gpk and catch's concerns in #30 and #31.

Here is are 2 patches. The first is a very simple one that uses the '*' wildcard syntax that is accepted by the major search engines (at least) to exclude just modules and themes. This fixes the immediate problem with uploaded files (e.g. PDFs) not being search-able.

The second is a more radical variant that opens things up to the degree that I think is appropriate. I can't see any reason for limiting access to the theme and modules directories, since this allows search engines the ability to index all the images that make up the page (and image search is *very* popular - it's still the #2 link of Google's tools), and I would not be surprised if some search engines are actually building a rough idea of the page layout behind the scenes, so they can better determine relevancy - and allowing images helps them better do that. If your PHP source code is visible then you have a serious security problem, so I don't think that is something to target here.

I think we can all agree on non radical patch so I am RTBC for that.

IMO, that does not matter. If a robot ever got there (how would it discover that URL? Very rare), they would see an empty page and probably not record it in their index. Whats your point? we already said that if you are disclosing your php you have bigger and out of scope problems.

I think we're happy with this one. It's been bike shedded. How changing robots.txt broke the tests I don't know.

um, #494462: Allow crawling of sites/default/files by search engines, don't disallow it in robots.txt (NIH)