Problem/Motivation

filter_html_image_secure currently uses file_url_transform_relative(), which only recognizes the host the request came in on. We are using the backport of this on Drupal.org, and it got us into some trouble, #1737022: Host Whitelist.

If the request comes in via Drush, there isn't a hostname. This may or may not apply in D8, since the method for getting the Request's hostname is completely different. The big problem is that the incorrect result is cached.

A more common problem is that sites can live at multiple hostnames. Drupal.org will be moving to www.drupal.org, a requirement for #2238131: Drupal.org CDN deployment notes. We should allow image URIs from either domain. Making https://drupal.org/… suddenly invalid for images would not be a nice thing to do.

Proposed resolution

Support a whitelist of domains in the filter plugin.

Remaining tasks

  • Question: Should this use trusted host names by default?
  • Needs tests including an update test as filter config will change.
  • Needs an update

User interface changes

Filter will have new configuration options and translated strings.

API changes

Internal function changes.

Data model changes

Filter plugin will have schema change.

Release notes snippet

TBD.

Original report by drumm

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

drumm’s picture

Status: Active » Needs review
FileSize
3.89 KB

Attached is an initial patch. I'm interested in knowing if this is a good approach. If so, we can proceed with any necessary changes and tests.

Getting a firm plan for this issue will let us proceed with the backport #1737022: Host Whitelist and hopefully get Drupal.org served via a CDN before DrupalCon Austin.

drumm’s picture

I should point out that "incorrect" values for allowed domains are not a problem. The URIs are rewritten to be relative, and checked against the filesystem, so they will serve correctly in the end.

sun’s picture

  1. +++ b/core/modules/filter/config/schema/filter.schema.yml
    @@ -69,6 +69,13 @@ filter_settings.filter_html:
    +    filter_html_image_secure_domains:
    +      type: string
    

    The key should just be "domains" and the type should be an array (which is type 'sequence', unless I'm mistaken)

  2. +++ b/core/modules/filter/filter.module
    @@ -1065,13 +1065,28 @@ function _filter_html_image_secure_process($text) {
    +      . implode('|', array_map('preg_quote',
    

    preg_quote() always needs the second parameter specified; i.e., the PCRE delimiter being used by the pattern.

  3. +++ b/core/modules/filter/filter.module
    @@ -1065,13 +1065,28 @@ function _filter_html_image_secure_process($text) {
    +          array_map('trim', preg_split('/(\r\n?|\n)/', $domains))
    

    This trim is obsolete after changing the configuration to an array.

  4. +++ b/core/modules/filter/filter.module
    @@ -1065,13 +1065,28 @@ function _filter_html_image_secure_process($text) {
         $src = $image->getAttribute('src');
         // Transform absolute image URLs to relative image URLs: prevent problems on
         // multisite set-ups and prevent mixed content errors.
    -    $image->setAttribute('src', file_url_transform_relative($src));
    +    if (isset($domains_regexp)) {
    +      $image->setAttribute('src', preg_replace($domains_regexp, '', $src));
    +    }
    +    else {
    +      $image->setAttribute('src', file_url_transform_relative($src));
    +    }
    

    With this change, I wonder whether we shouldn't switch this code entirely to use an approach that is based on parse_url(), so that we can simply check the parsed 'host' against the allowed domains?

drumm’s picture

Status: Needs review » Needs work

Agreed this can switch to parse_url(). That should make the code more readable.

Re #1: is there a good example elsewhere in core for the UI to go with an array/sequence on strings in configuration? Does the splitting on newlines happen elsewhere, or is there a matching form widget for multiple text fields?

drumm’s picture

FileSize
4.34 KB

Attached is an updated patch. Still needs feedback - is this a good approach?

drumm’s picture

Status: Needs work » Needs review
sun’s picture

This looks much better now. Mostly minor issues below, but a major task is to add (unit) test coverage for all possible permutations of $domains vs. $src. (I think we can skip test coverage for the UI configuration, so just focus on the filter processing.)

  1. +++ b/core/modules/filter/config/schema/filter.schema.yml
    @@ -69,6 +69,16 @@ filter_settings.filter_html:
    +  label: 'Filter HTML image secure'
    

    Hm, that's the filter's internal name only, which doesn't appear anywhere else as label.

    The plugin uses "Restrict images to this site"

    I don't know what the policy for such config schema plugin cases is...

  2. +++ b/core/modules/filter/config/schema/filter.schema.yml
    @@ -69,6 +69,16 @@ filter_settings.filter_html:
    +          label: 'Value'
    

    'Domain name', no?

  3. +++ b/core/modules/filter/filter.module
    @@ -1057,7 +1057,7 @@ function _filter_html_escape($text) {
    +function _filter_html_image_secure_process($text, $domains) {
    

    Let's type-hint $domains as an array.

  4. +++ b/core/modules/filter/filter.module
    @@ -1070,8 +1070,20 @@ function _filter_html_image_secure_process($text) {
    +      $url = parse_url($src);
    

    FYI: The backport to D7 will need error-suppression here; the PHP warning on malformed input was removed in PHP 5.3.3.

  5. +++ b/core/modules/filter/filter.module
    @@ -1070,8 +1070,20 @@ function _filter_html_image_secure_process($text) {
    +      // If the URL has a host, check if it is in the list of allowed domains.
    +      if (isset($url['host']) && isset($url['path']) && in_array($url['host'], $domains)) {
    

    Hm. Why is there no else condition?

    I also think we want to use a strict comparison for in_array() (i.e., 3rd param: TRUE).

  6. +++ b/core/modules/filter/lib/Drupal/filter/Plugin/Filter/FilterHtmlImageSecure.php
    @@ -25,8 +29,37 @@ class FilterHtmlImageSecure extends FilterBase {
    +      '#title' => $this->t('Alternate domains'),
    

    Shouldn't this be labeled "Allowed domains"?

  7. +++ b/core/modules/filter/lib/Drupal/filter/Plugin/Filter/FilterHtmlImageSecure.php
    @@ -25,8 +29,37 @@ class FilterHtmlImageSecure extends FilterBase {
    +      '#description' => $this->t('If your site reachable from muliple domains, such as example.com and www.example.com, list them one per line.'),
    

    As a user, it's not clear to me what happens if I don't fill in any domains.

    "If left empty, only image URLs that are equal to the current domain are allowed."

    (my addition reads a bit long and complex; let's find a way to shorten + simplify it)

  8. +++ b/core/modules/filter/lib/Drupal/filter/Plugin/Filter/FilterHtmlImageSecure.php
    @@ -25,8 +29,37 @@ class FilterHtmlImageSecure extends FilterBase {
    +      if ($configuration['settings']['domains'] === '') {
    

    This comparison should use a trim() on its own already, so that pure white-space is not interpreted as a value.

drumm’s picture

FileSize
4.43 KB

Attached is a patch with the corrections from #7.

Still needs new tests.

drumm’s picture

drumm’s picture

+++ b/core/modules/filter/filter.module
@@ -1070,8 +1070,20 @@ function _filter_html_image_secure_process($text) {
+      // If the URL has a host, check if it is in the list of allowed domains.
+      if (isset($url['host']) && isset($url['path']) && in_array($url['host'], $domains)) {

Hm. Why is there no else condition?

This follows the behavior of file_url_transform_relative(), a matching domain is converted to a relative URI. Either way, the next section of code checks that the image is in the filesystem. Non-relative URIs will fail that check.

m1r1k’s picture

Assigned: drumm » m1r1k
Issue tags: +Amsterdam2014
drumm’s picture

FileSize
2.27 KB

This could use a reroll.

Status: Needs review » Needs work

The last submitted patch, 12: 2257291.diff, failed testing.

The last submitted patch, 12: 2257291.diff, failed testing.

drumm’s picture

Status: Needs work » Needs review

I missed applying a chunk of this patch. The configuration UI is still missing.

(I'm using this issue to test #2518056: Need the ability to cancel a test job, so I'm setting this to Needs review so tests can at least start.)

drumm’s picture

Status: Needs review » Needs work

The last submitted patch, 16: 2257291.diff, failed testing.

The last submitted patch, 16: 2257291.diff, failed testing.

The last submitted patch, 16: 2257291.diff, failed testing.

mgifford’s picture

Assigned: m1r1k » Unassigned

The last submitted patch, 16: 2257291.diff, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mradcliffe’s picture

Version: 8.6.x-dev » 8.7.x-dev
Issue summary: View changes
Status: Needs work » Needs review
FileSize
3.31 KB

I ran into this recently as well. It's often not feasible to back port a gigantic filesystem of changing content between environments. It looks like I'm really more after #2825598: file_url_transform_relative() only works with current request's host, but found this issue first. I'm not sure if these issues should be combined or if that one should soley focus on file_url_transform_relative and this one on adding a whitelist of domains. I think for drupal.org's use case, fixing file_url_transform_relative would be fine?

We would need an update hook and update tests to test the configuration schema change.

I re-rolled @drumm's patch in #16 based on 8.7.x. There were not any differences detected by interdiff as a 3-way merge worked (curl https://www.drupal.org/files/issues/2257291_3.diff | git apply -3 --index).

This should be flipped back to Needs work after the test run if it doesn't fail.

Status: Needs review » Needs work

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

R_H-L’s picture

Patch in #28 fails in Drupal 8.8, both on existing and on a fresh install:

ResponseText: Doctrine\Common\Annotations\AnnotationException: [Semantical Error] Couldn't find constant domains, class Drupal\filter\Plugin\Filter\FilterHtmlImageSecure. in Doctrine\Common\Annotations\AnnotationException::semanticalError() (line 54 of /var/www/vendor/doctrine/annotations/lib/Doctrine/Common/Annotations/AnnotationException.php).

Edit: To clarify I got this during install.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.