Fixed a typo. Added a note that robots.txt is ineffective unless at the root of a host. Removed /files/ as a disallowed directory as I believe most people want Google to find their images and such.

CommentFileSizeAuthor
robot.diff1.12 KBjvandyk
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Dries’s picture

Status: Needs review » Fixed

Committed. Thanks.

Anonymous’s picture

Status: Fixed » Closed (fixed)
mfb’s picture

Version: x.y.z » 7.x-dev
Status: Closed (fixed) » Active

Re-opening this old bug to ask:
Would it also make sense to allow robots access to /sites/ since this is where the files directory will now be created?

Freso’s picture

Assigned: jvandyk » Unassigned

I'd be in favour of "no". But I'm also using private downloading, so...

However, if we're going to allow anything in /sites/ being indexed, we need to very specifically make it /sites/[all|default|*]/files. We don't want /sites/* or sites/[all|default|*]/* being indexed if they should somehow be accessible to the casual HTTP-reader.

Freso’s picture

Status: Active » Closed (fixed)

Hm. And seeing the age of this issue, I'm also thinking it should probably go back to being closed and have a new one opened for this specific discussion.

mfb’s picture

Version: 7.x-dev » 5.x-dev

Yes it should be a new issue, although it's not a problem for me so I didn't file one :)