By bramdriesen on
Change record status: 
Published (View all published change records)
Project: 
Drupal core
Introduced in branch: 
9.5.x
Introduced in version: 
9.5.0
Issue links: 
https://www.drupal.org/node/3271222
Description: 

Drupal's default robots.txt file (used for informing web crawlers what paths to not index) has been updated to disallow indexing of Oembed media links. This change only added following rules.

...
Disallow: /media/oembed
Disallow: /*/media/oembed
...
Disallow: /index.php/media/oembed
Disallow: /index.php/*/media/oembed
...

File changes:
- core/assets/scaffold/files/robots.txt
- robots.txt

Impacts: 
Site builders, administrators, editors