After update to new version of Boost 1.18 I got this error:
"Your robots.txt file does not contain "Disallow: /boost_stats.php". This is needed in order to prevent the output of this from accidentally getting indexed by search engines."

But I have the line Disallow: /boost_stats.php in my robots.txt (chmod 644).
What to do?

CommentFileSizeAuthor
#8 boost-707272.patch2.68 KBmikeytown2
robots.txt2.13 KBsuperfedya
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented
Title: Disallow: /boost_stats.php" entry missing in robots.txt » Disallow: /boost_stats.php" entry missing in robots.txt; use meta tag instead.
Category: bug » feature

robots.txt is only valid at the root level; I now check there.
example.com/robots.txt - Valid, change this one
example.com/drupal/robots.txt - Will not be read by a bot.
http://sitemaps.blogspot.com/2006/02/using-robotstxt-file.html

Now that I think about it, I should use the meta tag if possible.
http://www.mymommybiz.com/seo/metatagrobots.html
Will be hard to pull off. It might not be doable, since it embeds the html...

superfedya’s picture

superfedya CreditAttribution: superfedya commented

Ok, thanks, but I need write: Disallow: /boost_stats.php or Disallow: /mydrupalfolder/boost_stats.php?

mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented
Category: feature » bug

good point.
Disallow: /mydrupalfolder/boost_stats.php is what you want

superfedya’s picture

superfedya CreditAttribution: superfedya commented
Status: Active » Fixed

Thanks! Fixed.

mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented
Status: Fixed » Needs work

I need to put this fix in code

superfedya’s picture

superfedya CreditAttribution: superfedya commented

So, I changed this line Disallow: /boost_stats.php to Disallow: /mysite/boost_stats.php and got same error. Are You sure that its need be Disallow: /mysite/boost_stats.php and not just Disallow: /boost_stats.php?

mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented

Disallow: /mysite/boost_stats.php is correct; I need to put it in code so it will check the correct thing. To make it shut-up for now, do both.

mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented
Status: Needs work » Needs review
FileSize
boost-707272.patch2.68 KB

this should give correct info and check for the correct path if in a subdir

superfedya’s picture

superfedya CreditAttribution: superfedya commented

fixed.

mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented
Status: Needs review » Reviewed & tested by the community
mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented

committed

mikeytown2’s picture

mikeytown2 CreditAttribution: mikeytown2 commented
Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

yngens’s picture

yngens CreditAttribution: yngens commented
Status: Closed (fixed) » Active

The issue is actual for 6.x-1.20.

Path in # is giving:

patching file boost.install
Reversed (or previously applied) patch detected! Assume -R? [n] y
Hunk #1 succeeded at 72 (offset 8 lines).
Hunk #2 FAILED at 182.
Hunk #3 succeeded at 242 (offset 9 lines).
1 out of 3 hunks FAILED -- saving rejects to file boost.install.rej

bgm’s picture

bgm CreditAttribution: bgm commented
Status: Active » Fixed

@ yngens: the patch has been committed and it shipped with 6.x-1.20. There is no need to apply the patch.

Besides, the patch is mainly to clarify how to modify your robots.txt. You still need to modify it manually.

yngens’s picture

yngens CreditAttribution: yngens commented
Status: Fixed » Active

bgm, thanks for the comments, but before starting troubleshooting this and applying any patch I, of course, made sure to use the last version of the module. Could you please test this on a clean Drupal 6 install on CentOS machine to see the error message on status page is still there. My directory structure is as follows:

/home/username/public_html <---- and Drupal sits here, that is robots.txt is here: /home/username/public_html/robots.txt

and my robots.txt contains 'Disallow: /boost_stats.php'. To get rid of the message I tried all imaginable modifications of the path - the error marks doesn't want to go away. Is it only me or could anyone confirm with fresh Drupal install and the last version of the module?

bgm’s picture

bgm CreditAttribution: bgm commented

Can you copy-paste the complete error message?

bgm’s picture

bgm CreditAttribution: bgm commented
Status: Active » Closed (fixed)

Re-closing, no activity. Please open a new bug report if the problem persists and you have a complete error message.