Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Fixed a typo. Added a note that robots.txt is ineffective unless at the root of a host. Removed /files/ as a disallowed directory as I believe most people want Google to find their images and such.
Comment | File | Size | Author |
---|---|---|---|
robot.diff | 1.12 KB | jvandyk | |
Comments
Comment #1
Dries CreditAttribution: Dries commentedCommitted. Thanks.
Comment #2
(not verified) CreditAttribution: commentedComment #3
mfbRe-opening this old bug to ask:
Would it also make sense to allow robots access to /sites/ since this is where the files directory will now be created?
Comment #4
Freso CreditAttribution: Freso commentedI'd be in favour of "no". But I'm also using private downloading, so...
However, if we're going to allow anything in /sites/ being indexed, we need to very specifically make it /sites/[all|default|*]/files. We don't want /sites/* or sites/[all|default|*]/* being indexed if they should somehow be accessible to the casual HTTP-reader.
Comment #5
Freso CreditAttribution: Freso commentedHm. And seeing the age of this issue, I'm also thinking it should probably go back to being closed and have a new one opened for this specific discussion.
Comment #6
mfbYes it should be a new issue, although it's not a problem for me so I didn't file one :)