My apologies if this has already been reported.
I was using Google Webmaster Tools the other day and tried to submit an xml sitemap. I got an error that the robots.txt could not be downloaded (and hence they couldn't process the sitemap). I created a robots.txt file and uploaded it to the root and the sitemaps uploaded to fine after that. So, what I discovered is that Google doesn't like the dynamic robotos.txt in some instances. Since we have to delete the robotos.txt from the root for this module to work, I wanted to bring this to your attention. Perhaps it would make Google happier if this module were to write to/generate the sitemap file and leave it/place it in the root.
Thanks... I really like the module as is saves me from accidentally overwriting the robotos.txt, but the Google thing was an unpleasant surprise.
| Comment | File | Size | Author |
|---|---|---|---|
| #6 | robotstxt from GWT.gif | 11.03 KB | russellb |
Comments
Comment #1
hass commentedFix your broken webserver. NO robot have an idea that this file is not on your hardisks as it's always shown if requested from outside.
Comment #3
greenc commentedThere 's no problem with the web server.
Comment #4
hass commentedThan it would simply work.
Comment #5
greenc commentedHave you tried to test it with Google's webmaster tools?
Comment #6
russellb commentedI encountered a similar issue with Google & sitemaps in robotstxt 7.x-1.2. In Google Webmaster Tools, I was seeing robots.txt appearing and disappearing. I've attached a screenshot of the dropdown on the GWT robots.txt tester. XML sitemaps integration was running here.
I've switched the robotstxt module off and gone to a static robots.txt and the problem has gone away.
Comment #7
hass commentedComment #8
russellb commentedI've edited my post to make it clear that the problems I saw were with robotstxt 7.x-1.2.