DDEV is the official local development tool of Drupal. And like Drupal, DDEV depends on the support of the open source community.Problem/Motivation
it only affects those using a default theme
The default robots.txt uses Disallow: /core/ and under /core/ in Drupal8 we have the default Themes. This means a Searchengine can´t crawl the default Themes.
Proposed resolution
Google announced allowing Googlebot to retrieve JavaScript and CSS.
My advice would be to change the robots.txt like this.
User-agent: *
Crawl-delay: 10
# Directories
Disallow: /core/assets
Disallow: /core/includes
Disallow: /core/lib
Disallow: /core/misc
Disallow: /core/modules
Disallow: /core/profiles
Disallow: /core/scripts
Disallow: /core/test
Disallow: /core/vendor
Disallow: /profiles/
#Files in Core Dir
Disallow: /core/authorize.php
Disallow: /core/CHANGELOG.txt
Disallow: /core/core.services.yml
Disallow: /core/INSTALL.mysql.txt
Disallow: /core/INSTALL.pgsql.txt
Disallow: /core/install.php
Disallow: /core/phpunit.xml.dist
Disallow: /core/update.php
Disallow: /core/UPGRADE.txt
Disallow: /core/.gitignoreare you agree?
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | New_Robots_txt_Syntax_Checker__a_validator_for_robots_txt_files.png | 474.78 KB | askibinski |
| #6 | crawling-test.jpg | 60.89 KB | eule |
Comments
Comment #1
eule CreditAttribution: eule commentedComment #2
eule CreditAttribution: eule commentedComment #3
dawehnerDon't we actually want something like this? Just curious, but it seems odd to require disallowing so many entries.
Comment #4
eule CreditAttribution: eule commentednice thx @dawehner :-) looks cleaner
Comment #5
dawehnerSo yeah it would be great if you could provide a patch for that.
Comment #6
eule CreditAttribution: eule commentedNot a clue on how to provide a patch. Anyway i test it today and we have to change it with a slash in the front.
so let´s do this in this way will work!
Comment #7
eule CreditAttribution: eule commented@dawehner thats ok for you?
https://github.com/ekn33/drupal/commit/de4cb7081ad19eb3dd819eb35b16b3186...
Comment #8
askibinski CreditAttribution: askibinski commentedI'm seeing these "allow" rules are actually comitted here to Drupal 8. But they are
not valid according to Google webmastersand various syntax parser tools (see screenshot).edit: they seem to be allowed according to google's webmaster specs but they still give a syntax error... weird....
Comment #22
smustgrave CreditAttribution: smustgrave at Mobomo commentedThank you for creating this issue to improve Drupal.
We are working to decide if this task is still relevant to a currently supported version of Drupal. There hasn't been any discussion here for over 8 years which suggests that this has either been implemented or is no longer relevant. Your thoughts on this will allow a decision to be made.
Since we need more information to move forward with this issue, the status is now Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.
Thanks!
Comment #23
smustgrave CreditAttribution: smustgrave at Mobomo commentedWanted to bump this 1 more time.