Follow-up to #2364343: Fix robots.txt to allow search engines access to CSS, JavaScript and image files

Problem/Motivation

Google is starting to index JavaScript-rendered pages.

On 27th of October Google published new guidelines to ask for site owners to modify their robots.txt:

For optimal rendering and indexing, our new guideline specifies that you should allow Googlebot access to the JavaScript, CSS, and image files that your pages use. This provides you optimal rendering and indexing for your site. Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.

One of the most critical files (in Drupal 7) is the /misc/jquery.once.js file, that is not accessible to google with the default robots.txt. This stops most of javaScript code inside Drupal.behaviors not to be executed.

I tested myself and verify manually with Webmaster Tools that /misc and /modules css and JS files are blocked to Google.

This is mitigated by the fact that only affects to sites with CSS and JS Aggregation OFF. Once that is aggregated is enabled, the aggregated JSS and CSS lives in /files folder, that is accessible to Google.

Proposed resolution

Change robots.txt file

Remaining tasks

We need to rethink the best default robots.txt.

User interface changes

none

API changes

none

CommentFileSizeAuthor
#3 2546552-allow-google-access-to-css-and-js-d7-2.patch1.04 KBtikaszvince
allow-google-access-to-css-and-js-d7.patch389 bytestikaszvince

Comments

tikaszvince created an issue. See original summary.

tikaszvince’s picture

tikaszvince commented
Title: Fix robots.txt to allow Google access to CSS and JavaScript files » Fix robots.txt to allow Google access to CSS and JavaScript files [D7]
tikaszvince’s picture

tikaszvince commented
StatusFileSize
new 2546552-allow-google-access-to-css-and-js-d7-2.patch1.04 KB

I tested this deeper and found Allow css, jpg and image files do not enough to make these files reachable by Google bots.
So I changed the "Allow" rules to enable every type of these files placed under the disallowed directories.

mikemadison’s picture

mikemadison commented

Is there a reason you wouldn't also include themes in sites/all/themes so that custom themes' css and js are also included by default?

tikaszvince’s picture

tikaszvince commented

Because /sites is not in Disallow: rules we do not have to add it to Allow: rules.

neograph734’s picture

neograph734
he/him
Location Netherlands
commented
Title: Fix robots.txt to allow Google access to CSS and JavaScript files [D7] » Fix robots.txt to allow Google access to CSS and JavaScript files and image files [SEO][D7]
Priority: Major » Critical
Issue tags: +SEO

Marking this as critical, please correct me if I am wrong but I'd like to attract some maintainer attention here. After reading #2364343: Fix robots.txt to allow search engines access to CSS, JavaScript and image files this seems bigger than we might think.

This is potentially affecting lots and lots of Drupal 7 sites out there, because the forum and list icons are in /misc as well. So all sites using Bartik as theme (or another theme which does not explicitly override these images) are suffering negative SEO scores for over some time now.

The patch proposed here seems a bit too much, as there are no jpeg or svg images in /misc, so why should we allow them? (For installation profiles it might be useful though).
joegraduate's patch - comment #32 in #2364343: Fix robots.txt to allow search engines access to CSS, JavaScript and image files solves it like this:

diff --git a/robots.txt b/robots.txt
index ff9e286..6f58361 100644
--- a/robots.txt
+++ b/robots.txt
@@ -15,6 +15,22 @@
 
 User-agent: *
 Crawl-delay: 10
+# JS/CSS
+Allow: /misc/*.js
+Allow: /misc/*.css
+Allow: /modules/*.js
+Allow: /modules/*.css
+Allow: /profiles/*.js
+Allow: /profiles/*.css
+Allow: /themes/*.js
+Allow: /themes/*.css
+# Images
+Allow: /misc/*.gif
+Allow: /misc/*.png
+Allow: /profiles/*.gif
+Allow: /profiles/*.jpeg
+Allow: /profiles/*.jpg
+Allow: /profiles/*.png
 # Directories
 Disallow: /includes/
 Disallow: /misc/

But lets fix this as fast as we can please.

cilefen’s picture

cilefen commented
Priority: Critical » Major
Status: Needs review » Closed (duplicate)

Thank you for reporting this (again), but if I am not wrong this is a duplicate of #2364343: Fix robots.txt to allow search engines access to CSS, JavaScript and image files. See the backport policy. Please see the priority levels to find out this is not Critical.

neograph734’s picture

neograph734
he/him
Location Netherlands
commented

Understood, thanks.

But isn't this a much bigger issue for D7 sites than it is for D8? So is it possible to get something like at least a hotfix for this or something? I am aware of the backport policy, but that means this should be solved for D8 first before it gets backported (it is hotfixed there). This obviously makes sense, but it can take a while as there are bigger D8 issues to solve. Yet lots of sites suffer of bad SEO and there already is a solution.

I've found this issue and also wanted to raise some awareness amongst the D7 maintainers who might have missed this in D8 issue queue. It was in good faith, but won't happen again.

cilefen’s picture

cilefen commented

The other issue is a bug, so it could maybe go in 8.0.x, but read the discussion on that first. The best thing you could do now is post whatever patches are needed on the other issue. It is also acceptable to post a D7 patch there for people to test.

neograph734’s picture

neograph734
he/him
Location Netherlands
commented

Thank you, I read the other discussion and there is already a D7 patch in #32 there but it remained unnoticed. But I cannot RTBC that for D8. And switching versions in the issue also doesn't seem like a good idea.

cilefen’s picture

cilefen commented

I can't even determine the status of the main issue. Feel free to take control of that one to get it done if you understand the problems. If it is not quite finished for D8, send more patches. If it is finished, move it to 7.x-dev and send patches.

neograph734’s picture

neograph734
he/him
Location Netherlands
commented

Awesome, will do. Thanks for your time and guidance!