Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.Problem/Motivation
Fix robots.txt to allow search engines access to CSS, JavaScript and image files
Steps to reproduce
View the robots.txt which has allowed css, js files , images from core and profiles folder but the same is not the case for for modules and themes folder
Proposed resolution
To allow css, js and images under modules and themes folder.
#modules
Allow: /modules/*.css$
Allow: /modules/*.css?
Allow: /modules/*.js$
Allow: /modules/*.js?
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /modules/*.svg
#themes
Allow: /themes/*.css$
Allow: /themes/*.css?
Allow: /themes/*.js$
Allow: /themes/*.js?
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
Allow: /themes/*.svg
Remaining tasks
User interface changes
Introduced terminology
API changes
Data model changes
Release notes snippet
Similar issue was fixed in Drupal 7 : https://www.drupal.org/project/drupal/issues/2364343
But it also did not include sites/all/modules or sites/all/themes (as is the path for core and contrib modules/themes in drupal 7)
Was that done for security reasons ?
| Comment | File | Size | Author |
|---|---|---|---|
| #4 | Screenshot 2024-12-11 at 10.43.30 AM.png | 239.86 KB | anmolgoyal74 |
Issue fork drupal-3489435
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes
Comments
Comment #2
cilefen commentedThis seems to be a task or a feature request rather than a bug. robots.txt is editable by site operators.
Comment #3
saurav-drupal-dev commentedWorking
Comment #4
anmolgoyal74 commentedI think the CSS, JS and Images are allowed to be crawled. In D7, the modules, themes, profiles and misc directories are not not allowed to be crawled that's why it was add in robots.txt of D7. I think we can close this.
https://git.drupalcode.org/project/drupal/-/blob/7543eae758df91a217e6464...
Comment #6
stborchertComment #8
saurav-drupal-dev commentedI have updated the robot txt file please review.
Comment #9
saurav-drupal-dev commentedComment #10
smustgrave commentedThink we need some research first. Anyone check the history and see why these were added? Also this seems like something a project can do themselves. There's also https://www.drupal.org/project/robotstxt
Comment #11
poker10 commentedrobots.txt does not seems to block
/modulesor/themesdirectories, see: https://git.drupalcode.org/project/drupal/-/blob/11.x/robots.txt?ref_typ... . This is different from D7 behavior, which blocks these directories by default.I agree with #4 that there is probably nothing needed here. Feel free to reopen if you still think anything is needed, but will need an issues summary update in such case. Thanks!