By shaktik on
Change record status: 
Published (View all published change records)
Project: 
Drupal core
Introduced in branch: 
9.1.x
Introduced in version: 
9.1.0
Issue links: 
https://www.drupal.org/node/3123285
Description: 

Drupal's default robots.txt file (used for informing web crawlers what paths to not index) has been updated to disallow indexing of the user registration, password reset, login, and logout routes. These rules were already in place for years, but because they had a trailing slash, were ineffective. This change has removed that trailing slash.

Before:

  Disallow: /user/register/
  Disallow: /user/password/
  Disallow: /user/login/
  Disallow: /user/logout/

After:

  Disallow: /user/register
  Disallow: /user/password
  Disallow: /user/login
  Disallow: /user/logout

File changes:
- core/assets/scaffold/files/robots.txt
- robots.txt

Impacts: 
Site builders, administrators, editors