Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 UTC on 18 March 2024, to get $100 off your ticket.
By stefan.r on
Change record status:
Published (View all published change records)
Project:
Introduced in branch:
7.x
Introduced in version:
7.50
Issue links:
Description:
The robots.txt
file shipped with Drupal core has been modified to allow for CSS, JavaScript, and images located in misc/, modules/, profiles/, and themes/ to be indexed by search engines.
(The CSS and JavaScript changes only affect sites with CSS or JavaScript aggregation turned off.)
It is recommended that sites which maintain a customized robots.txt
file make similar changes in order to help search indexes better index the site.
Impacts:
Site builders, administrators, editors
Module developers
Comments
Reason ? Advantages ?
I fail to see why we would want CSS and Javascript files to be indexed by search engines (whether these files are compressed or not in the Drupal performance options).
As for images, they were already indexed by Google I believe, but I might add to robots.txt the exceptions for images inside /misc (only. Why would I want images in default /themes and modules to be indexed).
EDIT : the discussion/explanation is here. The Google bot already scans your /files folder where your CSS and Javascripts are aggregated/compressed.
Michael Lessard
webmaster of Quebec City "democracy in action" media