The comments in the robots.txt file in D6, D7 and D8 (haven't checked further back) all contain the following:

For more information about the robots.txt standard, see:
http://www.robotstxt.org/wc/robots.html

This url now returns a 404. The nearest equivalent seems to be http://www.robotstxt.org/robotstxt.html.

CommentFileSizeAuthor
#1 more-info-link-in-robots-txt-no-longer-valid-2290435-1-d8.patch389 bytespq

Comments

pq’s picture

pq commented
StatusFileSize
new more-info-link-in-robots-txt-no-longer-valid-2290435-1-d8.patch389 bytes

Patch attached

n.b. The comments block appears to be identical in D6/7/8, so the patch should apply to all.

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented
Status: Active » Reviewed & tested by the community
Issue tags: +Needs backport to D7, +Needs backport to D6

Looks right to me, thanks! [and see note above, patch should apply to all versions without any changes probably]

  • jhodgdon committed 8a44b5c on 8.x 
    Issue #2290435 by PQ: Fix broken link in robots.txt file

  • jhodgdon committed 7e6ba8f on 7.x 
    Issue #2290435 by PQ: Fix broken link in robots.txt file
jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented
Status: Reviewed & tested by the community » Fixed

Thanks again PQ! Committed to 8.x, 7.x, and 6.x.

  • jhodgdon committed 94bc7ef on 6.x 
    Issue #2290435 by PQ: Fix broken link in robots.txt file
pq’s picture

pq commented

Hi jhodgdon, glad to have helped

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.