On file robots.txt you can see:

# For syntax checking, see:
# http://www.frobee.com/robots-txt-check

And the link http://www.frobee.com/robots-txt-check not exist.

Note: There doesn't seem to be a good replacement for this. The lines just above this in the file direct you to robotstxt.org for more information; better that they maintain information about syntax and how to check it than that we do! So we should just remove the "For syntax checking" lines from our file, as these types of sites may come and go.

CommentFileSizeAuthor
#9 drupal7-2446657-9.patch333 byteser.pushpinderrana
#4 2446657-4.patch333 bytesrpayanm
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

amateescu’s picture

After a quick search, the only tool that I can find on a site that doesn't try to sell you something is: https://webmaster.yandex.com/robots.xml

But maybe we should just link to the canonical page for it, even if they don't really have a checker themselves: http://www.robotstxt.org/checker.html

jhodgdon’s picture

Hm. That canonical page has very little information on it. Perhaps the best thing is just to take out those two lines entirely, since probably these types of sites come and go. Just above, there is already a link to the base robotstxt.org site. That is probably good enough.

rpayanm’s picture

Assigned: Unassigned » rpayanm

Then working in this issue.

rpayanm’s picture

Assigned: rpayanm » Unassigned
Status: Active » Needs review
FileSize
333 bytes
jhodgdon’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community
Issue tags: -robots.txt, -robots, -robotstxt +Needs backport to D7

Thanks! [Updated summary to explain why we decided this was the right patch]

Also I think we will need to backport this patch to Drupal 7 once it is committed.

alexpott’s picture

alexpott’s picture

Version: 8.0.x-dev » 7.x-dev
Status: Reviewed & tested by the community » Patch (to be ported)

Committed 7bc8b51 and pushed to 8.0.x. Thanks!

  • alexpott committed 7bc8b51 on 8.0.x
    Issue #2446657 by rpayanm: Dead link on robots.txt
    
er.pushpinderrana’s picture

Status: Patch (to be ported) » Needs review
FileSize
333 bytes

Patch for D7.

amateescu’s picture

Status: Needs review » Reviewed & tested by the community

Looks good.

David_Rothstein’s picture

Status: Reviewed & tested by the community » Fixed

Committed to 7.x - thanks!

  • David_Rothstein committed a577cc9 on 7.x
    Issue #2446657 by rpayanm, er.pushpinderrana: Remove dead link from...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.