Problem/Motivation

Drupal's .htaccess and web.config files provide reasonable protection for sensitive files from prying eyes. There is FilesMatch (Apache) and a rewrite rule (IIS) that block accessing certain files based on a file name match. Looking at the history of the file over time, we only kept adding new patterns but never cleaned them up.

They provide protection for:
- Accessing PHP files directly via the web server (module, install, inc, profile, etc).
- Accessing sensitive static files (twig, yml, po, composer.json, composer.lock, etc).
- Access said files as temporary edit files of Vim and emacs (swo, swp, ~, etc).
- Patch residue (.bak, .orig, #foo.php#, etc).

Among these rules, there are some out dated rules such as code-style.pl that are no longer relevant (code-style.pl was removed many years ago).

The regular expression is getting difficult to read and there are some improvements that we can (micro) optimize.

Proposed resolution

Evaluate the current list of rewrite rules. Remove the ones that we no longer need, and combine/optimize them.

Remaining tasks

* Discuss on the components to remove as they are no longer necessary.
* Optimize the rewrite rules to adapt to modern directory structures (.well-known directory is excluded from ^\..* matching for example)
* Optimize the regular expression with non capturing group.
* Discuss to change the order of rewrite rules to make them easier to read (file extensions, followed by exact file names, followed by dot files, followed by edit/patch residue, etc).
* Remove file patterns that we block, but are no longer relevant. For example, SVN-related file matches are not necessary because a parent level dot-directory match will block them. Other than that, Entries.*|Repository|Root|Tag|Template|all-wcprops|entries|format seems to have survived from early as 2003, but I couldn't find concrete evidence why they should be blocked, considering we block the .svn directory they are contained in.
* Update .htaccess and web.config configuration.
* Update nginx and other web server documentation as necessary.

User interface changes

None.

API changes

None.
The release notes will need to say that we updated the .htaccess file, and the tests need to be updated of course. No other API changes.

Data model changes

None.

CommentFileSizeAuthor
#26 3017095-nr-bot.txt149 bytesneeds-review-queue-bot
#12 interdiff_9-12.txt696 bytestatarbj
#12 3017095-12-htaccess-regex.patch2.3 KBtatarbj
#9 3017095-9-htaccess-regex.patch2.3 KBayesh
#9 interdiff-8-9.patch2.13 KBayesh
#8 3017095-7-htaccess-regex.patch1.44 KBayesh
#4 3017095-4.patch1.68 KBchewie

Issue fork drupal-3017095

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

Ayesh created an issue. See original summary.

tatarbj’s picture

tatarbj
Primary language Hungarian
Location Brussels
commented
Issue tags: +ContributionWeekend2019
chewie’s picture

chewie commented
Assigned: Unassigned » chewie
Status: Active » Needs work
chewie’s picture

chewie commented
StatusFileSize
new 3017095-4.patch1.68 KB

Here is some patch with some optimization from my side.
I added some optimization for regexp.
For example removed in any cases access to trash/backup files in any cases.
Removed records about mysterious Entries.*|Repository|Root|Tag|Template|all-wcprops|entries|format. If you have any idea why this is necessary to keep this files in .htaccess/web.config, please, remind.

chewie’s picture

chewie commented

Currently, I don't have idea how to make regular expression more readable. It could be done for /.../x modifier. But unfortunately, for instance, Apache do not support PCRE standards (https://httpd.apache.org/docs/2.4/glossary.html#regex), otherwise it could be possible use syntax like this: https://softwareengineering.stackexchange.com/questions/194975/readable-...

From my side I think this is for now don't see big sense protect all possible files, which could contain sensitive data. It is more important to find solution how we could add possibility for developers to extend protections for files with sensitive data.

chewie’s picture

chewie commented
Assigned: chewie » Unassigned
Status: Needs work » Needs review
chewie’s picture

chewie commented
Status: Needs review » Needs work
ayesh’s picture

ayesh commented
Status: Needs work » Needs review
StatusFileSize
new 3017095-7-htaccess-regex.patch1.44 KB

Thanks for the awesome work! It looks like the tests failed due to emacs temp files (/~$/) not being blocked.
I have updated the match from @Chewie to exclude all files ending with a tilde. These files are almost always temporary files, and regardless of the extension, should be blocked. Current rule does not block all extensions with the tilde; just .module and .php~, which leaves other temp files (.install~, yml~, et al) accessible. Regex tester.

- Because using files with .lock ending is a pattern for lock files, I have moved the composer.lock rule to a generic .lock rule.

- Along with .lock extensions, one could think to block .pid files, but we'd have more problems if we there are any .pid files out in the open. I don' think we need to go that far.

- One more improvement would be making the rules case insensitive. For Apache-style FilesMatch, a capturing group that starts like ?i: makes it case insensitive. For IIS, the <match ignoreCase="false" /> does the trick.

ayesh’s picture

ayesh commented
StatusFileSize
new interdiff-8-9.patch2.13 KB
new 3017095-9-htaccess-regex.patch2.3 KB

This new one blocks all dot-files in the root except .well-known (that is used in acme protocol, Safari password change URL, etc). I also added tests for said files.

The last submitted patch, 9: interdiff-8-9.patch, failed testing. View results

ayesh’s picture

ayesh commented
tatarbj’s picture

tatarbj
Primary language Hungarian
Location Brussels
commented
Status: Needs review » Reviewed & tested by the community
StatusFileSize
new 3017095-12-htaccess-regex.patch2.3 KB
new interdiff_9-12.txt696 bytes

It seems good to me, only a small typo is fixed in my patch. I believe it's an RTBC as a clean-up issue.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 12: 3017095-12-htaccess-regex.patch, failed testing. View results

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

FatherShawn made their first commit to this issue’s fork.

fathershawn’s picture

fathershawn
Primary language English
Location New York
commented
Status: Needs work » Needs review

"re-rolled" these changes against 9.2.x

FatherShawn opened merge request !290

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

draenen’s picture

draenen commented

The block for Entries.* caused a problem on our site due to some PDF files starting with "Entries". Took awhile to track down why those PDFs were returning 403s so it would be nice to clean these up if they're not needed. Patch from #12 worked great.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

needs-review-queue-bot’s picture

needs-review-queue-bot commented
Status: Needs review » Needs work
StatusFileSize
new 3017095-nr-bot.txt149 bytes

The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.