updating robots.txt on a d6 boost website, a question came:
"do I need the "Disallow: /cache/" in robots.txt ?

Comments

marios88’s picture

marios88 commented

Probably not since no one knows you are using it ( provided you dont link it from anywhere on your site )

bgm’s picture

bgm commented
Status: Active » Fixed

I agree with marios88. You can still add it, but I have never had any issues with bots indexing the cache.

The installation documentation does not mention anything about it and the /cache has not changed in Boost for a long time, so I'm sure someone would have file a bug report if it were a problem.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.