robots.txt should disallow all '/search/' and '/search?' pages by default [#3550083]

Problem/Motivation

Drupal Core provides a robots.txt file that dissalows /search/.
Please note the trailing slash in that entry.
Link to source: https://git.drupalcode.org/project/drupal/-/blob/11.x/robots.txt?ref_typ...

Drupal sites often don't rely in core search but in custom views with filters or search_api views and facets.
Most of the times these use http query arguments, in the form of /search?title=robots.
Please note the absence of a trailing slash, the word search is followed by the arguments (?)

The current Drupal Core robots.txt rules do not apply for /search?

This creates a scenario where bots/crawlers, bad or legitimate like Googlebot and Bingbot will start indexing all search pages, creating a bad index and, in the case of facets, also causing server performance issues as they try to follow any possible combination of faceted search pages.

Steps to reproduce

- Install Drupal
- Create a search_api faceted search
- Submit site to google
- See crawl logs and see googlebot index faceted pages
- See it trying every possible combination, an endless situation

Proposed resolution

All of this can be easily fixed for all sites by changing:
Disallow: /search/
to

Disallow: /search/
Disallow: /search?

See https://developers.google.com/search/docs/crawling-indexing/robots/robot... for details and info on what led to this suggestion.

Remaining tasks

Fix the line in robots.txt, both in drupal root and scaffold files folder.

Release notes snippet

The default robots.txt file now blocks search result pages with query parameters from being crawled by search engines. Previously, only path-based search URLs (like /search/keyword) were blocked, but query parameter searches (like /search?q=keyword) remained accessible to crawlers. This caused search engines to index dynamically generated search results and crawl infinite combinations of faceted search pages, degrading site performance and SEO quality.

Site owners who have customized their robots.txt file should consider manually adding these rules:

Disallow: /search?
Disallow: /index.php/search?

This change only affects new installations and sites using the default scaffold robots.txt. Existing custom robots.txt files will not be automatically updated. See the change record (draft, not published) for more information.

Comment	File	Size	Author
#29	3550083-nr-bot__iwjpt52.txt	768 bytes	needs-review-queue-bot
#14	3550083-nr-bot_9rrwrlm_.txt	91 bytes	needs-review-queue-bot
#11	3550083-nr-bot_tjlc_6z6.txt	91 bytes	needs-review-queue-bot

Issue fork drupal-3550083

Show commands

Start within a Git clone of the project using the version control instructions.

Add & fetch this issue fork’s repository

Or, if you do not have SSH keys set up on git.drupalcode.org:

Add & fetch this issue fork’s repository

3 hidden branches

3550083-robotstxt-search-d10

changes, plain diff MR !13402

Check out this branch for the first time

Check out existing branch, if you already have it locally

3550083-robots.txt-search

changes, plain diff MR !13399

Check out this branch for the first time

Check out existing branch, if you already have it locally

3550083-

compare

Check out this branch for the first time

Check out existing branch, if you already have it locally

About issue forks

Comments

Comment #1

3 October 2025 at 07:57

bserem created an issue. See original summary.

Comment #2

3 October 2025 at 08:00

bserem opened merge request !13399

Comment #3

bserem commented 3 October 2025 at 08:01

Status:

Active

» Needs review

Comment #4

bserem commented 3 October 2025 at 08:08

Issue summary:

View changes

Comment #5

3 October 2025 at 10:30

bserem changed the visibility of the branch 3550083- to hidden.

Comment #6

3 October 2025 at 10:32

bserem opened merge request !13402

Comment #7

smustgrave commented 3 October 2025 at 14:14

Version:	11.2.x-dev	» 11.x-dev
Status:	Needs review	» Needs work

MRs should point to 11.x and should be one MR please.

May want to add an additional entry but not sure we should remove the existing. Temped to tag for framework manager sign off.

Comment #8

markconroy commented 3 October 2025 at 16:48

Thanks for working on this @bserem. I've seen a lot of LocalGov Drupal sites getting hammered because of crawlers following all facets pages.

I think this issue/MR would definitely qualify as a "Needs framework manager signoff" issue.

Comment #9

6 October 2025 at 06:56

bserem changed the visibility of the branch 3550083-robotstxt-search-d10 to hidden.

Comment #10

bserem commented 6 October 2025 at 07:02

Status:

Needs work

» Needs review

Thanks for the pointers @smustgrave

Updated the target branch
Hid the MR towards Drupal 10
I am not allowed to delete my branches from gitlab

Having an extra entry is also an option, I opted against it for less repetition, as the current change achieves the same results.

Putting back to needs review.

Comment #11

needs-review-queue-bot commented 6 October 2025 at 07:27

Status:

Needs review

» Needs work

Status	File	Size
new	3550083-nr-bot_tjlc_6z6.txt	91 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Comment #12

bserem commented 14 October 2025 at 13:25

Status:

Needs work

» Needs review

Recreated the branch and the MR to not include any 11.2.x stuff as advised in #7 above.

The diff is now clean. Marking as needs review again.

Comment #13

smustgrave commented 14 October 2025 at 14:11

Issue tags:

+Needs framework manager review

Comment #14

needs-review-queue-bot commented 14 October 2025 at 19:32

Status:

Needs review

» Needs work

Status	File	Size
new	3550083-nr-bot_9rrwrlm_.txt	91 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Comment #15

nod_

French

Lille

commented 14 October 2025 at 20:32

Status:

Needs work

» Needs review

Comment #16

bserem commented 16 October 2025 at 08:23

Issue summary:

View changes

Comment #17

ifrik

she/her

commented 17 October 2025 at 09:28

Status:	Needs review	» Reviewed & tested by the community
Issue tags:		+Vienna2025

This works, and it solves the issue. Core's search pages use the path /search/node and /search/user so the trailing / make sense, but other search pages (for example when building a search view using Search API) often just use /search.
Since the core search paths are still covered by this change, it's a simple change that will benefit other setups as well.

Comment #18

poker10 commented 18 October 2025 at 22:48

Status:

Reviewed & tested by the community

» Needs work

Thanks for working on this.

I think that the proposed change can cause issues on some sites. As per https://developers.google.com/search/docs/crawling-indexing/robots/robot... , the Disallow: /search will match anything starting with /search, so it has a potential to also block any content with URL alias starting with that string (for example a news with an alias of /search-for-donations, etc.). Based on this, moving back to NW.

Also this is probably more like a Feature request than a Bug, given there is no issue with the core itself.

Comment #19

bserem commented 19 October 2025 at 13:05

Assigned:

Unassigned

» bserem

Comment #20

bserem commented 20 October 2025 at 06:51

Status:

Needs work

» Needs review

Comment #21

bserem commented 20 October 2025 at 06:51

Assigned:

bserem

» Unassigned

Comment #22

bserem commented 26 October 2025 at 17:01

I've only now realized that comments submitted in gitlab might not be visible here according to user settings. Work has been done in https://git.drupalcode.org/project/drupal/-/merge_requests/13399#note_60... and screenshots have been added in that comment to visualize the changes.

Quoting gitlab comment here, without screenshots, for clarity:

Thanks @poker10 for the review. Your concern is valid, so I updated the robots.txt file to disallow /search/ (as it was) and /search? (new entry).

Comment #23

bserem commented 30 October 2025 at 13:42

Title:	robots.txt should disallow all '/search*' pages by default	» robots.txt should disallow all '/search/' and '/search?' pages by default
Issue summary:	View changes

Comment #24

smustgrave commented 19 November 2025 at 19:24

Status:

Needs review

» Reviewed & tested by the community

Believe feedback has been addressed.

Comment #25

catch

he/him

English

commented 12 January 2026 at 18:11

Status:	Reviewed & tested by the community	» Needs work
Issue tags:		+Needs change record, +Needs release note

This is a good idea but it needs a change record and release notes snippet since it's changing site owner managed/scaffolded files.

Comment #26

bserem commented 13 January 2026 at 09:54

Issue summary:	View changes
Status:	Needs work	» Reviewed & tested by the community

Made a change record: https://www.drupal.org/node/3567198
Added "Release notes snippet".

Full transparency: as English is not my first language I did use an LLM to improve my writing to something more formal.

Moving this back to RTBC, I hope I this is the right place.

Comment #27

bserem commented 13 January 2026 at 09:57

Issue tags:

-Needs change record, -Needs release note

Cleaning up tags.

I also feel "needs framework manager review" can go since @smustgrave and @catch have checked this.

Comment #28

13 January 2026 at 09:57

Version:

11.x-dev

» main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Comment #29

needs-review-queue-bot commented 31 January 2026 at 05:32

Status:

Reviewed & tested by the community

» Needs work

Status	File	Size
new	3550083-nr-bot__iwjpt52.txt	768 bytes

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Comment #30

smustgrave commented 31 January 2026 at 15:33

Status:

Needs work

» Reviewed & tested by the community

Bot rebellion

feat: #3550083 robots.txt should disallow all '/search/' and '/search?'...

Comment #34

13 March 2026 at 17:30

godotislate committed 519d5d64 on 11.x

feat: #3550083 robots.txt should disallow all '/search/' and '/search?'...

Comment #35

godotislate

he/him

commented 13 March 2026 at 17:31

Version:	main	» 11.x-dev
Category:	Bug report	» Feature request
Status:	Reviewed & tested by the community	» Fixed
Issue tags:	-Needs framework manager review