This patch removes the obsolete paths /database/ and /update/ and reorders the other paths and files by alphabet.

This patch needs to be backported to D6 and D5 as both also contains the obsolete path's.

CommentFileSizeAuthor
D7_updated_robotstxt.patch1.04 KBhass
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Dries’s picture

Version: 7.x-dev » 6.x-dev

I've committed this to CVS HEAD. Thanks. Lowering version to Drupal 6.

hass’s picture

Status: Needs review » Reviewed & tested by the community

Patch also applies cleanly to D6 and D5.

Gábor Hojtsy’s picture

I am a bit reluctant to commit this patch. On the plus side, it might prevent disallowing indexing of directories, the user might create in her Drupal installation, such as /database and /updates, so that might be an improvement for Drupal 6. On the minus side, it is unlikely that these paths are used and robots.txt is often edited, so we should reserve to changing it on production branches only when required, not for "cosmetic" changes. Otherwise we just give people some headache to diff and update their robots.txt for the new version without much benefit.

Gábor Hojtsy’s picture

alexanderpas’s picture

Title: Updated robotxt.txt » Updated robots.txt
Priority: Normal » Minor
Gábor Hojtsy’s picture

Version: 6.x-dev » 5.x-dev

Ok, committed to Drupal 6 to coincide with the .htaccess change as well at: http://drupal.org/node/28776

drumm’s picture

Version: 5.x-dev » 6.x-dev
Status: Reviewed & tested by the community » Fixed

Committed to 5.x.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.