By james2002 on

I have installed drupal to following website.

http://www.sciencenewsheadlines.com

But I wonder should I block google Search engine to index from mysite/node and onwards to avoid duplicated content being indexed as I am also using Url alias.

thanks

⋅ Categories: Drupal 6.x

Comments

gausarts’s picture

gausarts commented

Yes, place that in robots.txt and consider the following also if you use modules. Google consumes lots of bandwidth for what we don't actually want it to.

User-agent: *
Disallow: /admin
Disallow: /aggregator
Disallow: /tracker
Disallow: /node/add
Disallow: /user
Disallow: /files
Disallow: /search
Disallow: /book/print
Disallow: /filter
Disallow: /filter/tips
Disallow: /comment
Disallow: /comment/reply
Disallow: /spam
Disallow: /spam/report
Disallow: /spam/report/comment
Disallow: /node/

User-agent: *
Crawl-delay: 10
Disallow: /aggregator/
Disallow: /tracker/
Disallow: /comment/reply/
Disallow: /node/add/
Disallow: /user/
Disallow: /privatemsg/
Disallow: /mail/
Disallow: /files/
Disallow: /book/print/
Disallow: *?page=
Disallow: *?from=
Disallow: /taxonomy/
Disallow: /archive/

Check your logs for more and perhaps, multilingual paths

love, light n laughter
blogid.net

love, light n laughter

james2002’s picture

james2002 commented

Thanks
What is crawl delay?

Do you put robot.txt in root of drupal?

Web hosting sites