With the extensive use of letsencrypt and acme challenges does robots.txt file need Disallow: \. to exclude .well-known directory?

Comments

izmeez created an issue. See original summary.

aaronbauman’s picture

Status: Active » Closed (won't fix)

No.
.well-known is used for more than just acme cert challenges, including many assets that are meant to be publicly accessible to crawlers.

See https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml

izmeez’s picture

Thank you.