Include Disallow Oembed media links in the robots.txt file for better Drupal SEO

Problem/Motivation

Search engines are indexing the media/oembed links in websites
And the search results for the oembed remote media can be viewed in search results
Then a landing page will open with only the content of the remote media in it.

Include Oembed media link in the robots.txt to be disallowed

# Oembed media
Disallow: /media/oembed
Disallow: /en/media/oembed
Disallow: /fr/media/oembed
Disallow: /es/media/oembed
Disallow: /ar/media/oembed

This is a repetitive issue that the support team faces when starting the handover process

Drupal core robots.txt file dose not have the Disallow: /media/oembed
and only following with http://www.robotstxt.org/robotstxt.html

Proposed resolution

Add a line “Disallow: /media/oembed“ to be configured from Drupal core to be always included in robots.txt
Regarding the other languages will be dealt with by the delivery team to add the variation according to the available languages on the website

Cover both single and multilingual sites.

Disallow: /media/oembed
Disallow: /*/media/oembed

And clean and not clean Urls

Disallow: /index.php/media/oembed
Disallow: /index.php/*/media/oembed

Remaining tasks

• ✅ File an issue
• ✅ Addition/Change/Update/Fix
• ✅ Testing to ensure no regression
• ✅ Automated unit testing coverage
• ✅ Automated functional testing coverage
• ➖ UX/UI designer responsibilities
• ➖ Readability
• ➖ Accessibility
• ✅ Performance
• ✅ Security
• ➖ Documentation
• ✅ Code review by maintainers
• ✅ Full testing and approval
• ➖ Credit contributors
• ➖ Review with the product owner
• ✅ Release notes snippet
• ❌ Release

User interface changes

• N/A

API changes

• N/A

Data model changes

• N/A

Release notes snippet

• Drupal's default robots.txt file (used for informing web crawlers what paths to not index) has been updated to disallow indexing of Oembed media links.