Support RFC 5785 by whitelisting the .well-known directory

The patch supplied with the issue was not enough or didn't work. Attaching a rule that works for me.

Sorry ignore the above patch, it was created against D7, this one has modification particular to D8

@mfb, maybe it could also be noted that its not necessary to create the ".well-known" directory. It could be handled by drupal through a hook_menu declaration.

This patch works as expected.

Let's Encrypt (https://letsencrypt.org/) uses .well-known for its verification process. Without this patch Drupal users cannot generate free SSL/TLS certificates.

The #4 patch is applying without errors and working fine. So I set the Status to "RTBC".

Already existing Drupal projects with individual settings in their .htaccess file needs a manual change with these settings.
And because this can easily done manually and D6 is near EOL we don't not need a D6 backport and I remove this tag.
But I add the tag "Quick fix" because It's just a minor change in the default .htaccess file.

Here is a D7-backport patch which needs review to prepare the next step. Because it's against D7 the automatic test should fail.

I'm currently failing with starting a retest on D7 branch.
For now I reset the issue back to D8 and RTBC for patch "issue-2408321-support-rfc5785-04.patch".

This affects security in that it updates the regexp of protected files/directories in .htaccess.

My suggestion:

D7:
\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(?!\.well-known\/acme-challenge)(\..*|Entries.*|Repository|Root|Tag|Template)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig\.save)$

D8:
\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(?!\.well-known\/acme-challenge)(\..*|Entries.*|Repository|Root|Tag|Template)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$

Nginx users can allow the .well-known directory like this (above the general line to block hidden directories and other stuff):

# Allow Let's Encrypt RFC 5785 ACME protocol
location ~* ^/.well-known/ {
  allow all;
}

Submitted a pull request @ https://github.com/nginxinc/nginx-wiki/edit/master/source/start/topics/recipes/drupal.rst which was accepted earlier.

Just to be clear, Let's Encrypt is not the only practical use case. For example, CalDAV and CardDAV also depend on .well-known.

thanks @walterebert, sent a pull request to Nginx Wiki which removes reference to Let's Encrypt, now only references "Well-Known URIs".

Updated Nginx code in IS too.

We looked at this in the Frankfurt sprint.

This could use a change record for 8.1.x - sites that have already got a custom .htaccess could easily miss the change, and a change notice at least lets people checking those do the extra step. CNR for that.

I created a draft change record at: https://www.drupal.org/node/2661732 .

Change record looks fine.

This changes ^(\..*|........)$ to ^(\.(?!well-known)|........)$ - why remove the .*? Seems that would remove protection for *all* dotfiles (note ^.(?!anything)$ is equivalent to ^.$). Granted, that makes no difference as long as mod_rewrite is enabled, thanks to the other rule, but still.

Applied the patch and tested the following scenarios:

mkdir .well-known
GET .well-known: Access forbidden

mkdir .not-well-known
GET .not-well-known: Access forbidden

echo 'test' > .well-known/test.txt
GET .well-known/test.txt: Access granted

echo 'test' > .not-well-known/test.txt
GET .not-well-known/test.txt: Access forbidden

rm -Rf .well-known
rm -Rf .not-well-known

Created a module with a .well-known route: Access granted
GET .well-known: Access granted

Created a module with a .not-well-known route: Access denied
GET .not-well-known: Access denied

Created a module with a .well-known/test.txt route: Access granted
GET .well-known/test.txt: Access granted

Created a module with a .not-well-known/test.txt route: Access denied
GET .not-well-known/test.txt: Access granted

This looks good, change to RTBC.

Committed/pushed to 8.1.x, thanks!

Moving to 7.x for backport.

Drupal 7 support would be very welcome. I'm not really sure how to submit a PR on D.org, but it should be pretty simple change, right?

<FilesMatch "\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig\.save)$">
  Order allow,deny
</FilesMatch>

  RewriteRule "(^|/)\.(?!well-known)" - [F]

Yep, pretty simple port

Applied the patch to D7 and tested the following scenarios:

mkdir .well-known
GET .well-known: Access forbidden

mkdir .not-well-known
GET .not-well-known: Access forbidden

echo 'test' > .well-known/test.txt
GET .well-known/test.txt: Access granted

echo 'test' > .not-well-known/test.txt
GET .not-well-known/test.txt: Access forbidden

rm -Rf .well-known
rm -Rf .not-well-known

Created a module with a .well-known route: Access granted
GET .well-known: Access granted

Created a module with a .not-well-known route: Access denied
GET .not-well-known: Access denied

Created a module with a .well-known/test.txt route: Access granted
GET .well-known/test.txt: Access granted

Created a module with a .not-well-known/test.txt route: Access denied
GET .not-well-known/test.txt: Access denied

From my point of view this is RTBC.

Should there be tests for this?

This is a patch for D7, the D8 patch has already been committed.
So, I'd say the testing is a followup?

Followup at #2699701: Testing RFC 5785 Support

Be good to see this committed for D7

RTBC + 1 for D7, marking for commit soon

Is it necessary to address https://www.drupal.org/node/2661732#comment-11000005 here first (and for Drupal 8 too)? That looks a bit iffy... although maybe not a big deal in practice.

By the way, that change notice is still in Draft state... I didn't publish it, due to the above - and also because it seems incomplete (it only refers to one half of the changes from this issue, which I don't think works on its own... probably better to point people to an actual diff if we're going to show people suggested changes to make).

I think we should fix the bug first in D8. Having the wrong things white-listed is bad.

Thanks David for careful review of the rules.

Adding an issue credit for the person who discovered that problem.

What's the procedure to get this committed to 7.x soon? It has been committed to the 8.x branches and there is already a patch for D7, so which conditions have not been met?

7.x is on hold for now because we have to fix it in 8.x first.
The next step is to create a patch for 8.x addressing the feedback from above.

I applied the changes committed to 8.3.x on a Drupal 7 .htaccess file.

Which concerns do we have that prevents a commit for Drupal 7?

The regex was wrong, so we have to fix D8 first.

If someone can do a patch, we can get this committed to 8.2.x pretty fast - I think AND then also to 7.x.

Attached patches for D8 and D7.

Hm, it feels to me that the OR is missing a (), too.

e.g. could someone do some manual regex testing on this?

I am using Drupal 7 and the changes in https://www.drupal.org/node/2408321#comment-11614247

It works fine on my site. The .well-known is publicly readable. And, contrary to the first version I tried earlier, when I redirect my site to only HTTPS, now I get the proper redirection; the previous version redirect http to a 403 error, caused by a [F] (or forbidden) in the error log.

Any other tests that I could do?

Not sure I can do the regex tests, though. :-)

I can test this if somebody could point me at a documentation of what is allowed and what is not allowed.

Defining Well-Known Uniform Resource Identifiers (URIs)
https://tools.ietf.org/html/rfc5785

I tested this on D7 as I needed it for Let'sEncrypt certificates. Patch seems to work for D7. D8 not tested.
But for D7 I would say RTBC.

oh wait, not just well-known uris @sanduhrs, but since same regex is used to protect other files and folders, i would like to test regression too. So what is Drupal's policy in general for what files should be shown and what not

I've been using this patch on D7 sites for a little while now, no problems. Can we please get it committed to 7.x?

I used the patch on #53 on my D7 site and now I get this error message:

The challenge url is not working with an HTTP status of 0. Please check your .htaccess as it may need modifications, see Support RFC 5785 by whitelisting the .well-known directory.

Before the patch I was getting this error:

The challenge url is not working with an HTTP status of 403. Please check your .htaccess as it may need modifications, see Support RFC 5785 by whitelisting the .well-known directory

To answer #55 (by a visual review, I haven't tested it) it looks right to me that there is no () on the OR. The first branch is just /\. which should catch any dotfile in a subdirectory. The second branch is all of ^\.(?!well-known/) i.e. the well-known exception applies only after ^\. which is not in a subdirectory. That is correct:

A well-known URI is a URI [RFC3986] whose path component begins with
the characters "/.well-known/", and whose scheme is "HTTP", "HTTPS",
or another scheme that has explicitly been specified to use well-
known URIs.

Why exactly is this issue still open? It has been committed to 8.2 and 8.3 after all.
I created a backport issue against 7.x to get it committed there.

For those of you using Virtualmin, don't forget that you also need to edit the ssl.conf file on your server for everything to work properly.

As per #56

@sanduhrs your updated patch hasn't been committed to D8 yet ;-)

Finally got around to testing this.

# check() { curl -I "http://localhost$1" 2>/dev/null | head -n 1; }
# check /
HTTP/1.1 200 OK
# check /.well-known
HTTP/1.1 403 Forbidden
# check /.well-known-not
HTTP/1.1 403 Forbidden
# mkdir -p .well-known/acme-challenge
# echo test > .well-known/acme-challenge/test
# check /.well-known/acme-challenge/test
HTTP/1.1 200 OK
# mkdir -p .well-known-not/acme-challenge
# echo test > .well-known-not/acme-challenge/test
# check /.well-known-not/acme-challenge/test
HTTP/1.1 403 Forbidden

Seems fine to me.

Committed/pushed to 8.4.x and cherry-picked to 8.3.x. Thanks!

Please open the 7.x backport.

The backport issue is already open: https://www.drupal.org/node/2847325
It just needs another set of eyes to change the status and a commit.

Published the CR at https://www.drupal.org/node/2661732. It probably fell under the radar because it was not associated with any project…

The change notice was never published because it was incorrect - see https://www.drupal.org/node/2408321#comment-11361857

I will try to fix it now.

Did my best to fix it via https://www.drupal.org/node/2661732/revisions/view/10527162/10527685

If someone wants to copy this and create a corresponding change record for the Drupal 7 version, feel free. But we already mentioned it in the 7.55 release notes so I didn't think it was that important.