Follow-up to #2297711: Fix HTML escaping due to Twig autoescape

Follow-up to #1825952: Turn on twig autoescape by default

Non-technical explanation

After #2264041: Add a test to ensure title callbacks are not vulnerable to XSS have proven that even battle hardened core developers can't write XSS free code we have introduced #1825952: Turn on twig autoescape by default to fix a torrent of security holes already present in core known and unknown and to avoid the most frequent kind of sechole(Security Hole) in the history of Drupal contrib. However, this has broken some places that were already securely written, resulting in broken layout and HTML tags shown to users. We need to find those places and update them to be compatible with the new method.

Problem/Motivation

Can be tricky to discover the double escaping.

Proposed resolution

@dawehner's and @joelpittet's idea about testing existing checked routes that are already tested for double escaping.
Duckpatch drupalGet/drupalPost methods in simpletest to check for double escaping on pages.
Potentially a more permanent fixture could be possible.

Remaining tasks

CommentFileSizeAuthor
#23 2571065.11-reroll.patch12.72 KBjoelpittet
#11 2571065.11.patch12.77 KBalexpott
#2 find_escaping_due_to-2571065-2.patch2.41 KBjoelpittet

Comments

joelpittet created an issue. See original summary.

joelpittet’s picture

joelpittet
he/him
Primary language English
Location Vancouver
commented
Issue summary: View changes
Status: Active » Needs review
Issue tags: -Perfomance, -Needs issue summary update
StatusFileSize
new find_escaping_due_to-2571065-2.patch2.41 KB

Status: Needs review » Needs work

The last submitted patch, 2: find_escaping_due_to-2571065-2.patch, failed testing.

The last submitted patch, 2: find_escaping_due_to-2571065-2.patch, failed testing.

dawehner’s picture

dawehner
Primary language German
commented

I know that I had an issue about that too, well, no idea where it is now :)

There have been just a few actual valid failures, most the other ones seems to be real bugs ... wow

Status: Needs work » Needs review

joelpittet queued 2: find_escaping_due_to-2571065-2.patch for re-testing.

joelpittet’s picture

joelpittet
he/him
Primary language English
Location Vancouver
commented

160 was the count, retesting to see where we are at so far.

Status: Needs review » Needs work

The last submitted patch, 2: find_escaping_due_to-2571065-2.patch, failed testing.

Status: Needs work » Needs review

joelpittet queued 2: find_escaping_due_to-2571065-2.patch for re-testing.

joelpittet’s picture

joelpittet
he/him
Primary language English
Location Vancouver
commented
Issue tags: +rc target triage

Would be nice to have no unintentional escaping with D8 release.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
StatusFileSize
new 2571065.11.patch12.77 KB

Combines #2597608: User role action labels get double-escaped and out of sync with the corresponding user role label to see what is left...

Status: Needs review » Needs work

The last submitted patch, 11: 2571065.11.patch, failed testing.

joelpittet’s picture

joelpittet
he/him
Primary language English
Location Vancouver
commented

32 to 6 is quite nice!

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented

@joelpittet it is even better than that! I'd argue we only really have to investigate 2 of the fails...

  • Drupal\Tests\user\Unit\Plugin\Action\RemoveRoleUserTest - not related solved by latest patch on the other issue
  • Drupal\filter\Tests\FilterAdminTest - expected fail the filter admin page has intentional double escaping
  • Drupal\search\Tests\SearchNodePunctuationTest - needs investigation
  • Drupal\simpletest\Tests\SimpleTestTest (this accounts for 2 of the fails) - needs investigation - but Simpletest UI so very very unimportant and this test is evil
  • Drupal\system\Tests\Theme\EntityFilteringThemeTest - needs investigation
alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented

Drupal\search\Tests\SearchNodePunctuationTest is fine - it is searching for & so it is an expected fail.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Related issues: +#2598502: Double escaping in views attachment titles

Created #2598502: Double escaping in views attachment titles to fix Drupal\system\Tests\Theme\EntityFilteringThemeTest

The last submitted patch, 2: find_escaping_due_to-2571065-2.patch, failed testing.

The last submitted patch, 2: find_escaping_due_to-2571065-2.patch, failed testing.

xjm’s picture

xjm
she/her
Primary language English
commented
Issue tags: -rc target triage

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

joelpittet’s picture

joelpittet
he/him
Primary language English
Location Vancouver
commented
Status: Needs work » Needs review
StatusFileSize
new 2571065.11-reroll.patch12.72 KB

Reroll

Status: Needs review » Needs work

The last submitted patch, 23: 2571065.11-reroll.patch, failed testing.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

smustgrave commented
Status: Needs work » Postponed (maintainer needs more info)
Issue tags: +stale-issue-cleanup

Thank you for creating this issue to improve Drupal.

We are working to decide if this task is still relevant to a currently supported version of Drupal. There hasn't been any discussion here for over 8 years which suggests that this has either been implemented or is no longer relevant. Your thoughts on this will allow a decision to be made.

Since we need more information to move forward with this issue, the status is now Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.

Thanks!

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

smustgrave’s picture

smustgrave commented
Status: Postponed (maintainer needs more info) » Needs work
Issue tags: +Needs issue summary update

This one appears to still be valid.