CSS and JS cannot be crawled by robots.txt; decreases SEO

Google have announced that they will rank mobile-friendly sites higher (implying by comparison that non-mobile-friendly sites will be penalised).

Google describe 3 mobile-friendly techniques :
1. Responsive Web Design
2. Dynamic Serving (changing CSS/JS resources etc depending on user-agent)
3. Serving mobile content on a different URL (e.g. m.example.com).

In order to analyse for Responsive Web Design, Google say:

"Google’s algorithms should be able to automatically detect this setup if all Googlebot user agents are allowed to crawl the page and its assets (CSS, JavaScript, and images)."

The current robots.txt disallows /themes and /profiles, which means that Google's crawler will not be able to successfully analyse core themes, or custom themes that are part of an installation profile. This means that sites using themes in this way will be penalised as not being mobile-friendly.

Note that this is mitigated when the core aggregation mecanism is enabled because CSS files will go to the sites/* folder, which is not excluded by the robots.txt file.

Moreover, this won't happen in non fine-tuned installations where the static files are excluded by the HTTPd, such as when using some Nginx default configuration files that we can find all around the place (when the robots.txt gives a 404 error).