Ignore crawling at google index for Juicebox XML

Hey there,

I've been using the following in robots.txt, and it seems to do the trick:

Disallow: /juicebox/xml/

This makes me wonder however if it would make sense to add some native controls on the module to help exclude the XML from SERPs. I can't image too many cases where someone would actually want the XML to be indexed. In this regard we can't really control the Drupal robots.txt programmatically, and <meta> tags can't be used on XML. However, we probably could set a X-Robots-Tag: noindex header when the XML is sent. I don't think this is as broadly supported by search engines as robots.txt or meta tags, but I think the big ones (Google, Yahoo, etc.) do support it.

It would be interesting to know what others think about this.

Since I personally use a robots.txt entry this concern has flown totally under my radar. I think it makes sense to have some sane default restrictions on this enforced by the module. X-Robots-Tag: noindex would probably be a good approach.

I'm adjusting metadata appropriately, though I'm not sure when I'll get to this. In the meantime also note that all Drupal users have rights to edit the documentation should someone want to add robot.txt notes there.

Cool, thanks for making that edit.

I'll look into the idea of adding a X-Robots-Tag: noindex header (hopefully soon). I believe the mechanics for that can work just like the handling of the Access-Control-Allow-Origin: * header control that was added for CORS support in #2171715: Avoid cross-domain restrictions if using Drupal as endpoint for gallery XML.

This change in now in dev for D7. The XML now uses an X-Robots-Tag: noindex header and I've also adjusted the content type header to application/xml (instead of text/xml) as I think this will also help crawlers identify the XML as content that's not intended for humans. I thought about making this X-Robots-Tag configurable in the settings, but I really can't think of any cases where someone would want to turn this off, and I suppose adding related options may just prove confusing.

This still needs to be addressed in 8.x-2.x however....

Ok, the relevant pieces have now been ported to 8.x-2.x.