Change record status: 
Project: 
Introduced in branch: 
7.x
Introduced in version: 
7.50
Description: 

The robots.txt file shipped with Drupal core has been modified to allow for CSS, JavaScript, and images located in misc/, modules/, profiles/, and themes/ to be indexed by search engines.

(The CSS and JavaScript changes only affect sites with CSS or JavaScript aggregation turned off.)

It is recommended that sites which maintain a customized robots.txt file make similar changes in order to help search indexes better index the site.

Impacts: 
Site builders, administrators, editors
Module developers
Updates Done (doc team, etc.)
Online documentation: 
Not done
Theming guide: 
Not done
Module developer documentation: 
Not done
Examples project: 
Not done
Coder Review: 
Not done
Coder Upgrade: 
Not done
Other: 
Other updates done

Comments

Michael_Lessard_micles.biz’s picture

I fail to see why we would want CSS and Javascript files to be indexed by search engines (whether these files are compressed or not in the Drupal performance options).

As for images, they were already indexed by Google I believe, but I might add to robots.txt the exceptions for images inside /misc (only. Why would I want images in default /themes and modules to be indexed).

EDIT : the discussion/explanation is here. The Google bot already scans your /files folder where your CSS and Javascripts are aggregated/compressed.

Michael Lessard
webmaster of Quebec City "democracy in action" media