PHP 5.4 introduces a new flag to htmlentities() "which makes htmlentities()/htmlspecialchars() replace the invalid multibyte sequences with U+FFFD (UTF-8) or &#xFFFD (other encodings).

In a nutshell, all the invalid character sequences are replaced by "�" instead of discarding the whole thing. That would make debugging way easier in many cases where data is fed into Drupal in non-UTF8 (by the database incorrectly returning non-UTF8 errors, by system extensions, on data import, etc. examples of this type of issue are numerous).

Twig's autoescape uses this internally.
https://github.com/twigphp/Twig/blob/641090378dfff2913929306fbbf227269ba...

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Task because it improves DX and consistency with Twig's escape
Issue priority Major because our Twig escaping will produce different results than Html::escape flags
Unfrozen changes Unfrozen because it only changes markup.
Prioritized changes The main goal of this issue is security and consistency
Disruption Not disruptive as it will be doing what Twig is already doing and invalid characters will encode into something visible
Files: 
CommentFileSizeAuthor
#28 1211866-28.patch5.5 KBstefan.r
PASSED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,244 pass(es). View
#28 interdiff-21-28.txt3.18 KBstefan.r
#24 1211866-21.patch4.66 KBstefan.r
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,222 pass(es), 5 fail(s), and 0 exception(s). View
#24 interdiff-20-21.txt2.88 KBstefan.r
#23 1211866-20.patch1.51 KBstefan.r
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,204 pass(es), 9 fail(s), and 0 exception(s). View
#22 interdiff-19-20.txt445 bytesstefan.r
#22 interdiff-19-20.txt445 bytesstefan.r
#21 1211866-19.patch1.51 KBstefan.r
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,216 pass(es), 10 fail(s), and 0 exception(s). View
#21 interdiff-18-19.txt694 bytesstefan.r
#18 1211866-18.patch1.51 KBstefan.r
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,228 pass(es), 5 fail(s), and 0 exception(s). View
#18 interdiff-17-18.txt415 bytesstefan.r
#17 interdiff-13-17.txt1.45 KBstefan.r
#17 1211866-17.patch1.71 KBstefan.r
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,195 pass(es), 10 fail(s), and 0 exception(s). View
#13 1211866-13.patch1.48 KBstefan.r
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,239 pass(es), 5 fail(s), and 0 exception(s). View
#8 enable_ent_substitute-1211866-8.patch3.05 KBjoelpittet
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 96,889 pass(es), 5 fail(s), and 0 exception(s). View
#5 ent_substitute-1211866-5.patch641 bytestsphethean
FAILED: [[SimpleTest]]: [MySQL] 57,997 pass(es), 19 fail(s), and 0 exception(s). View

Comments

Damien Tournoud’s picture

Issue tags: +PHP 5.4

Tagging.

sun’s picture

If this works reliably, then it would be even be a major reason for bumping PHP requirements to 5.4.

DamienMcKenna’s picture

I wouldn't recommend mandating PHP 5.4 be required for D8, but this definitely could be a good optional feature for when PHP 5.4 is available.

tsphethean’s picture

Do we have a baked in means of detecting PHP version, or would something like

    if (defined('ENT_SUBSTITUTE')) {
      $replace_flags = ENT_QUOTES & ENT_SUBSTITUTE;
    }
    else {
      $replace_flags = ENT_QUOTES;
    }

be appropriate?

tsphethean’s picture

Status: Active » Needs review
FileSize
641 bytes
FAILED: [[SimpleTest]]: [MySQL] 57,997 pass(es), 19 fail(s), and 0 exception(s). View

On the assumption the above is ok i've rolled this patch. Tests pass on my local on php 5.3.25 and 5.4.15 - not entirely sure what to do about writing additional unit tests to cater for this. Do we want php version checking logic in our unit tests, or do we just case that the cases are what we expect?

The last submitted patch, ent_substitute-1211866-5.patch, failed testing.

Anonymous’s picture

Issue summary: View changes

Fix preview

joelpittet’s picture

Component: base system » theme system
Category: Feature request » Task
Issue summary: View changes
Issue tags: +Twig

PHP 5.4 is already a requirement and Twig already does this so it would also keep things in line with Twig's escape filter.

https://github.com/twigphp/Twig/blob/1.x/lib/Twig/Extension/Core.php#L1011

+1 to this issue.

joelpittet’s picture

Status: Needs work » Needs review
FileSize
3.05 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 96,889 pass(es), 5 fail(s), and 0 exception(s). View

Re-rolled.

joelpittet’s picture

Status: Needs review » Needs work

The last submitted patch, 8: enable_ent_substitute-1211866-8.patch, failed testing.

alexpott’s picture

Status: Needs work » Postponed
Related issues: +#2550945: Add Html::escape()

This issue seems like a really good idea. Postponing on #2550945: Add Html::escape()

joelpittet’s picture

FYI twig is using this internally so another reason to have this.
https://github.com/twigphp/Twig/blob/641090378dfff2913929306fbbf227269ba...

stefan.r’s picture

FileSize
1.48 KB
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,239 pass(es), 5 fail(s), and 0 exception(s). View
stefan.r’s picture

Issue tags: -Needs tests
stefan.r’s picture

Status: Postponed » Active
alexpott’s picture

Status: Active » Needs work
+++ b/core/lib/Drupal/Component/Utility/Html.php
@@ -371,7 +371,8 @@ public static function decodeEntities($text) {
+   * including "&eacute;" and "&lt;" to "é" and "<", converting invalid
+   * multi-byte sequences to the Unicode Replacement Character ("�").

Let's not add this documentation to the bit about how this method is different from Html::decodeEntities. It belongs in the list of conversions above.

stefan.r’s picture

Status: Needs work » Needs review
FileSize
1.71 KB
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,195 pass(es), 10 fail(s), and 0 exception(s). View
1.45 KB

The list is a list of entity equivalents though, if we want it to be in the list we could drop the " with their HTML entity equivalents" bit?

stefan.r’s picture

FileSize
415 bytes
1.51 KB
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,228 pass(es), 5 fail(s), and 0 exception(s). View

a newline crept in

alexpott’s picture

alexpott’s picture

Status: Needs review » Needs work
+++ b/core/lib/Drupal/Component/Utility/Html.php
@@ -366,7 +366,8 @@ public static function decodeEntities($text) {
+   * (for example, "&lt;" becomes "&amp;lt;"), and invalid UTF-8 encoding
+   * will be converted to the Unicode Replacement Character ("�").

I think it should be "replacement character" - too many capitals :). I like where these docs are now. Thanks.

stefan.r’s picture

Status: Needs work » Needs review
FileSize
694 bytes
1.51 KB
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,216 pass(es), 10 fail(s), and 0 exception(s). View
stefan.r’s picture

FileSize
445 bytes
445 bytes
stefan.r’s picture

FileSize
1.51 KB
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,204 pass(es), 9 fail(s), and 0 exception(s). View
stefan.r’s picture

FileSize
2.88 KB
4.66 KB
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,222 pass(es), 5 fail(s), and 0 exception(s). View

Fixing some more test fails

The last submitted patch, 17: 1211866-17.patch, failed testing.

The last submitted patch, 18: 1211866-18.patch, failed testing.

joelpittet’s picture

Awesome! Thanks for jumping on this after the HTML::escape blocker was in.

stefan.r’s picture

FileSize
3.18 KB
5.5 KB
PASSED: [[SimpleTest]]: [PHP 5.5 MySQL] 103,244 pass(es). View

clarifying some comments

The last submitted patch, 21: 1211866-19.patch, failed testing.

The last submitted patch, 23: 1211866-20.patch, failed testing.

The last submitted patch, 24: 1211866-21.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 28: 1211866-28.patch, failed testing.

Status: Needs work » Needs review

stefan.r queued 28: 1211866-28.patch for re-testing.

The last submitted patch, 13: 1211866-13.patch, failed testing.

joelpittet’s picture

Status: Needs review » Reviewed & tested by the community

Thank you, green means go.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs beta evaluation

Normal tasks need a beta evaluation.

joelpittet’s picture

Title: Enable ENT_SUBSTITUTE in check_plain » Enable ENT_SUBSTITUTE flag in Html::escape
Priority: Normal » Major
Issue summary: View changes
Status: Needs work » Reviewed & tested by the community
Issue tags: -Needs beta evaluation
alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed 00360b9 and pushed to 8.0.x. Thanks!

Thanks for adding the beta evaluation to the issue summary.

  • alexpott committed 00360b9 on 8.0.x
    Issue #1211866 by stefan.r, joelpittet, tsphethean: Enable...
Wim Leers’s picture

Wow, this landed fast! Great job, @stefan.r :)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.