As mentioned by Dave Reid at http://drupal.org/node/1376398#comment-5382194, it would be great for Facet API to integrate with the RobotsTxt module. Facet API is in a great position to integrate with RobotsTxt because it knows the search path of each implementing module via the FacetapiAdapter::getSearchPath() method.

I don't see this as an addition to the core module, but rather a small contrib bridging the two modules. It would depend on Facet API and implement hook_robotstxt() to disallow each searcher's FacetapiAdapter::getSearchPath() paths. It could also get more complex and provide an admin interface allowing users to configure the robots.txt behavior for each searcher.