Hey Folks,
I recently made some changes to robots.txt in accordance with the SEO For Drupal 6 book, and, since then, Google Webmaster tools shows I have 532 HTTP errors (from zero prior!), all of which point to my users. I've since added disallow: /users to the robots.txt, but, with each passing day, Google adds another few hundred links to user profiles (which lead to a brick wall as no one's authorized to view them save for me). My question is, do I need to add/change anything elsewhere to get Googlebot to stop spidering/indexing this area of my site? Other than this error, I've cleaned up the site quite nicely, so I'm really desperate to get this resolved.
Here's my robots.txt file for your consideration. Have I done anything wrong here? Is my disallow:/users in the right place? Is there some sort of root command I need to enter to stop this from happening as well?? My site had been really getting beat up by Google, and I'd made so much progress. To have this happen is just heartbreaking :(
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /sites/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /contact
Disallow: /logout
Disallow: /node/add
Disallow: /search/
Disallow: /user/register
Disallow: /user/password
Disallow: /user/login
Disallow: /user
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=contact/
Disallow: /?q=logout/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Thanks in advance!!
Jim
Comments
Comment #1
horrorview commentedOkay, seems the robots.txt changes did nothing. I checked webmaster tools this morning and it found even more users/errors. >:(
Comment #2
chrismiller627 commentedYou may consider removing the offending URLs with this: https://www.google.com/webmasters/tools/removals and see if google stops spidering them in the future...
Comment #3
kars-t commentedHi
I am closing this issue to clean up the issue queue as it is very old. Feel free to reopen the issue.