Problem/Motivation

it only affects those using a default theme
The default robots.txt uses Disallow: /core/ and under /core/ in Drupal8 we have the default Themes. This means a Searchengine can´t crawl the default Themes.

Proposed resolution

Google announced allowing Googlebot to retrieve JavaScript and CSS.

My advice would be to change the robots.txt like this.

User-agent: *
Crawl-delay: 10
# Directories
Disallow: /core/assets
Disallow: /core/includes
Disallow: /core/lib
Disallow: /core/misc
Disallow: /core/modules
Disallow: /core/profiles
Disallow: /core/scripts
Disallow: /core/test
Disallow: /core/vendor
Disallow: /profiles/
#Files in Core Dir
Disallow: /core/authorize.php
Disallow: /core/CHANGELOG.txt
Disallow: /core/core.services.yml
Disallow: /core/INSTALL.mysql.txt
Disallow: /core/INSTALL.pgsql.txt
Disallow: /core/install.php
Disallow: /core/phpunit.xml.dist
Disallow: /core/update.php
Disallow: /core/UPGRADE.txt
Disallow: /core/.gitignore

are you agree?

Comments

eule’s picture

Issue summary: View changes
eule’s picture

Issue summary: View changes
dawehner’s picture

Don't we actually want something like this? Just curious, but it seems odd to require disallowing so many entries.

Allow: core/themes
Allow: core/*.css
Allow: core/*.js
Allow: core/misc
eule’s picture

nice thx @dawehner :-) looks cleaner

dawehner’s picture

So yeah it would be great if you could provide a patch for that.

eule’s picture

FileSize
60.89 KB

Not a clue on how to provide a patch. Anyway i test it today and we have to change it with a slash in the front.

so let´s do this in this way will work!

Allow: /core/themes
Allow: /core/*.css
Allow: /core/*.js
Allow: /core/misc

Webmastertools Crawling Report

eule’s picture

askibinski’s picture

I'm seeing these "allow" rules are actually comitted here to Drupal 8. But they are not valid according to Google webmasters and various syntax parser tools (see screenshot).

edit: they seem to be allowed according to google's webmaster specs but they still give a syntax error... weird....

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.