On file robots.txt you can see:
# For syntax checking, see:
# http://www.frobee.com/robots-txt-check
And the link http://www.frobee.com/robots-txt-check not exist.
Note: There doesn't seem to be a good replacement for this. The lines just above this in the file direct you to robotstxt.org for more information; better that they maintain information about syntax and how to check it than that we do! So we should just remove the "For syntax checking" lines from our file, as these types of sites may come and go.
| Comment | File | Size | Author |
|---|---|---|---|
| #9 | drupal7-2446657-9.patch | 333 bytes | pushpinderchauhan |
| #4 | 2446657-4.patch | 333 bytes | rpayanm |
Comments
Comment #1
amateescu commentedAfter a quick search, the only tool that I can find on a site that doesn't try to sell you something is: https://webmaster.yandex.com/robots.xml
But maybe we should just link to the canonical page for it, even if they don't really have a checker themselves: http://www.robotstxt.org/checker.html
Comment #2
jhodgdonHm. That canonical page has very little information on it. Perhaps the best thing is just to take out those two lines entirely, since probably these types of sites come and go. Just above, there is already a link to the base robotstxt.org site. That is probably good enough.
Comment #3
rpayanmThen working in this issue.
Comment #4
rpayanmComment #5
jhodgdonThanks! [Updated summary to explain why we decided this was the right patch]
Also I think we will need to backport this patch to Drupal 7 once it is committed.
Comment #6
alexpott#180379: Fix path matching in robots.txt is related
Comment #7
alexpottCommitted 7bc8b51 and pushed to 8.0.x. Thanks!
Comment #9
pushpinderchauhan commentedPatch for D7.
Comment #10
amateescu commentedLooks good.
Comment #11
David_Rothstein commentedCommitted to 7.x - thanks!