Hello,

I'm running a multisite setup. Found your (all of you) module. I've configured version 6.x-1.2 with the default robots.txt (+ minor tweaks) for a number of sites (not all). According to http://www.sxw.org.uk/computing/robots/check.html the syntax is OK.

That said, Google is still crawling sites & modules directories. See log entries sample below.


warning page not found 2010/10/26 - 04:53 sites/all/modules/faq/faq.css Anonymous
warning page not found 2010/10/26 - 04:53 modules/comment/comment.css Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/modules/ubercart/uc_attribute ... Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/modules/ubercart/uc_product/u ... Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/modules/ubercart/uc_store/uc_ ... Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/files/ctools/css/b3fc79f71f9c ... Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/themes/pixture_reloaded/layou ... Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/files/css_injector_5.css Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/modules/calendar/jcalendar/jc ... Anonymous
warning page not found 2010/10/26 - 04:53 sites/ahmstlambert.com/files/color/pixture_reloaded- ... Anonymous
warning page not found 2010/10/26 - 04:53 modules/system/system-menus.css Anonymous
warning page not found 2010/10/26 - 04:53 sites/all/modules/cck/modules/fieldgroup/fieldgroup.css Anonymous

Any suggestions?

Thanks
--> http://ahmstlambert.com/robots.txt

Comments

hass’s picture

hass commented
Status: Active » Fixed

This is not module specific. Google may crawl your full site, but it does not index the files as you have this files excluded. With a robots.txt you do not tell google don't crawl my site in this paths - no you only say do not show files in the index. This is documented somewhere on Google...

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.