Problem/Motivation

This is coming from #1665818: Ampersands and other characters are not displayed correctly in <select> widgets in the Entity reference queue but the bug reported there can be found in core now that ER is in core.

When implementing hook_options_list() and passing the options through check_plain(), any options with ampersands or < and > symbols end up double encoded in select widgets.

Quoting Damien Tournoud in #1665818-19: Ampersands and other characters are not displayed correctly in <select> widgets:

Core does a strip_tags() without a decode_entities()

Proposed resolution(s)

Drupal 7:
A) #38 (no D7 patch yet, this is what was committed to D8)
B) #18 (no patch yet, differs from what was committed to D8)
C) #48 (sent back in #57)
D) #58 (sent back in #63)

Drupal 8:
Add decode_entities() to the strip_tags block in _options_prepare_options().

Remaining tasks

  • Needs tests.
  • Update taxonomy module so the term reference field does a check_plain() in its hook_options_list() implementation?

User interface changes

n/a

API changes

TBD

Comments

star-szr’s picture

Issue summary: View changes

Tweaking remaining tasks formatting

star-szr’s picture

StatusFileSize
new556 bytes

Status: Needs review » Needs work

The last submitted patch, 1919338-1.patch, failed testing.

damien tournoud’s picture

       if ($properties['strip_tags']) {
          [...]
       }
       if ($properties['filter_xss']) {
          [...]
       }

This whole function needs to be refactored, because there are only two possible cases, not four. A simple $properties['html'] (true/false) would be enough.

Also, I usually recommend trim(str_replace("\n", " ", drupal_html_to_text($value))) as a way to down-case HTML to plain-text, because it results in a plain-text string that translates more of the intention of the HTML (for example, <em>text</em> is going to be transformed into *text*).

star-szr’s picture

Status: Needs work » Needs review

#1: 1919338-1.patch queued for re-testing.

JvE’s picture

Status: Needs review » Active

Would it not be easier to remove the check_plain()s from form_select_options() rather than

  1. encode in hook_options_list()
  2. decode in _options_prepare_options()
  3. encode in form_select_options()

?

damien tournoud’s picture

@JvE: no, it would not. Select elements require plain-text.

JvE’s picture

The current encoding flow for taxonomy select is:

  1. encode in form_select_options()


The current encoding flow for entityreference select is:

  1. encode in hook_options_list()
  2. encode in form_select_options()

Leading to double encoding.


This patch changes that to:

  1. encode in hook_options_list()
  2. decode in _options_prepare_options()
  3. encode in form_select_options()

and suggests changing taxonomy select to do the same.

damien tournoud’s picture

Title: Select widget prone to double encoding » Select widget (from the options module) prone to double encoding
Status: Active » Needs work

@JvE: yes, what you describe is the intended flow.

hook_options_list() specifies that its values are HTML:

The HTML tags defined in _field_filter_xss_allowed_tags() are allowed, other tags will be filtered.

In most cases, we already have HTML. For example, the allowed values for the field types managed by the list module are HTML.

But both Entity Reference (when using the base selection plugin) and Taxonomy Reference only have a plain-text label, so they need to transform it to HTML by using check_plain().

Inside the options module, we have two cases:

  • For checkboxes and radio buttons widgets, we can use the HTML we just need to pass it to filter_xss() for sanitization
  • For select widgets, we need to downcase the HTML into plain-text so that we can use that as a #options for the select widget

This last process is where the bug is. The current code only does a strip_tags() without a decode_entities(), which result in broken output. I recommend we change this to trim(str_replace("\n", " ", drupal_html_to_text($value))) so as to generate a plain-text value that keeps some of the intend of the original HTML.

sun’s picture

There's a third option, which is to finally get rid of the inconsistency of #options with regard to #type checkboxes, radios, and select:
#342316: Introduce proper Form API #types for 'option' and 'optgroup', and make #options consistent.

damien tournoud’s picture

@sun, yes, that makes sense, but that's not backportable. Let's come up with a solution we can actually use in 7.x.

star-szr’s picture

This probably needs be reconsidered now that #1751234: Convert option widgets to Plugin system is in. Is this fixed in 8.x now, and if so can we move this to 7.x?

catch’s picture

That patch doesn't change anything functionally, here's the relevant hunk:

+  /**
+   * {@inheritdoc}
+   */
+  static protected function sanitizeLabel(&$label) {
+    // Select form inputs allow unencoded HTML entities, but no HTML tags.
+    $label = strip_tags($label);
+  }
catch’s picture

Status: Needs work » Needs review
StatusFileSize
new1.99 KB
new1.31 KB

Here's a patch, test-only, then test with fix for Drupal 8.

catch’s picture

Version: 8.x-dev » 7.x-dev
StatusFileSize
new3.12 KB
new1.26 KB

Adding Drupal 7 backport, moving version temporarily for the bot.

JvE’s picture

Yes, I like it.

catch’s picture

Version: 7.x-dev » 8.x-dev
Issue tags: +Needs backport to D7

Back to 8.x again.

star-szr’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll

D8 patch in #13 looks good to me and has tests. Unfortunately it needs a reroll now.

damien tournoud’s picture

In #8, I suggested using trim(str_replace("\n", " ", drupal_html_to_text($value))) instead, because it keeps more of the semantics of the original HTML. Opinions on this?

rob230’s picture

I've used #14 on my D7 site. It fixes the double encoding and I've not had any problems so far.

catch’s picture

@Damien the strip_tags() is already there, this just fixes the double-encoding bug so I don't see any reason to change the intended behaviour here, might be worth opening another issue though.

shnark’s picture

StatusFileSize
new1.96 KB

Reroll.
I followed these instructions. https://drupal.org/patch/reroll.
There were no conflicts that I had to solve by hand:
$ git rebase 8.x
First, rewinding head to replay your work on top of it...
Applying: applying patch
Using index info to reconstruct a base tree...
M core/modules/options/lib/Drupal/options/Plugin/field/widget/SelectWidget.php
M core/modules/options/lib/Drupal/options/Tests/OptionsWidgetsTest.php
Falling back to patching base and 3-way merge...
Auto-merging core/modules/options/lib/Drupal/options/Tests/OptionsWidgetsTest.php
Auto-merging core/modules/options/lib/Drupal/options/Plugin/field/widget/SelectWidget.php

I looked at the patch from comment #13. The changes in that are similar looking to the changes in the reroll.

shnark’s picture

Status: Needs work » Needs review
shnark’s picture

Issue tags: -Needs reroll
star-szr’s picture

#21: 1919338_21.patch queued for re-testing.

ptvv’s picture

hook_options_list() specifies that its values are HTML

Surely one of the big problems here is precisely hook_options_list()'s documentation. If you include the previous sentence too the quote is

The labels should NOT be sanitized, options.module takes care of sanitation according to the needs of each widget. The HTML tags defined in _field_filter_xss_allowed_tags() are allowed, other tags will be filtered.

If the return values are supposed to be HTML, then they should be HTML-encoded already and the first sentence is incorrect: the only sanitisation that should be necessary is anti-XSS filtering, which should be idempotent, so rather than saying that the labels should "NOT" be sanitised it should say that it is not necessary to perform anti-XSS filtering on them.

On the other hand, if the return values are supposed to be plain text then no HTML tags should be allowed.

Either way, the documentation is at best misleading and at worst is actively encouraging developers to create bugs.

mustanggb’s picture

Been running the D7 port without any problems.

mustanggb’s picture

Issue summary: View changes

Remove unneeded comment number

wiifm’s picture

After reading the whole of #1665818: Ampersands and other characters are not displayed correctly in <select> widgets and then finding this issue, I applied @catch's patch at #14 and now my ER select list is no longer double encoded (Drupal 7).

+1 for that.

jamesdixon’s picture

D7 patch #14 worked great for me on Drupal 7.23.

I had entity reference fields with select widgets showing html entities incorrectly, but patch #14 cleared that up.

Thanks @catch and everyone who contributed to the fix.

bblake’s picture

Issue summary: View changes
StatusFileSize
new3.13 KB

Rolled against 7.22

Status: Needs review » Needs work

The last submitted patch, 29: options_1919338_D7_29.patch, failed testing.

star-szr’s picture

Status: Needs work » Needs review

#21 is the patch to review for D8.

damien tournoud’s picture

If the return values are supposed to be HTML, then they should be HTML-encoded already and the first sentence is incorrect: the only sanitisation that should be necessary is anti-XSS filtering, which should be idempotent, so rather than saying that the labels should "NOT" be sanitised it should say that it is not necessary to perform anti-XSS filtering on them.

That's just because you confuse "encoding" (which the action of transforming plain-text into HTML), with "sanitization" (which is the action of making sure that the HTML doesn't contain executable code). The documentation is misleading, but it is technically correct.

jwilson3’s picture

Patch in #14 also solves the problem visible in Drupal 7 Organic Groups select list on node edit forms for og-enabled content types, when an ampersand appears in a Group name.

webadpro’s picture

Path #14 did the trick for me also. When will this be pushed in the core for D7?

dooleysarahe’s picture

+1 patch #14 for D7. It would be great to see this committed to core.

alauddin’s picture

#14 patch for D7.26 as well.

mustanggb’s picture

Issue tags: +Novice

Tagging novice for a re-roll.

swentel’s picture

StatusFileSize
new1.93 KB

rerolled

andypost’s picture

Status: Needs review » Reviewed & tested by the community

finally, let's get this in! #13 has "fail-test"

alexpott’s picture

Version: 8.x-dev » 7.x-dev
Status: Reviewed & tested by the community » Patch (to be ported)

Committed 966c330 and pushed to 8.x. Thanks!

bradjones1’s picture

Status: Patch (to be ported) » Needs review
StatusFileSize
new1.26 KB
new3.13 KB

Rerolled #14 for D7.

The last submitted patch, 41: options_1919338_D7_41_test_only.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 41: options_1919338_D7_41.patch, failed testing.

bradjones1’s picture

Status: Needs work » Needs review
StatusFileSize
new1.26 KB
new3.12 KB

Let's try this again.

The last submitted patch, 44: options_1919338_D7_44_test_only.patch, failed testing.

bradjones1’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -Needs tests

No longer needs tests. I'm going to RTBC as I'm not the original author of the patch.

jwilson3’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/modules/field/modules/options/options.test
@@ -233,6 +233,7 @@ class OptionsWidgetsTestCase extends FieldTestCase {
     $this->assertRaw('Some dangerous &amp; unescaped markup', 'Option text was properly filtered.');
+    $this->assertRaw('Some HTML encoded markup with &lt; &amp; &gt;');

It wouldn't hurt to write an assertion message for the second test, to make it clear what you're testing for. Something like 'HTML entities in option text were properly handled and not double-encoded'.

Jalandhar’s picture

Status: Needs work » Needs review
StatusFileSize
new3.2 KB

Updating the patch as said in comment #47, Please review.

wiifm’s picture

Status: Needs review » Reviewed & tested by the community

Patch looks good to me, and fixes a fairly major bug with Drupal core at present. +1 RTBC

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 48: options_drupal7-1919338-48.patch, failed testing.

Jalandhar’s picture

Status: Needs work » Needs review
Jalandhar’s picture

I was not sure why the test had failed earlier but it got passed now after re-testing...:P

wiifm’s picture

Status: Needs review » Reviewed & tested by the community

Back to RTBC due to the testbot having a bad day.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 48: options_drupal7-1919338-48.patch, failed testing.

Jalandhar’s picture

Status: Needs work » Needs review
Jalandhar’s picture

Status: Needs review » Reviewed & tested by the community

Back to RTBC.

David_Rothstein’s picture

Status: Reviewed & tested by the community » Needs review

I don't think we should change all those property names or merge them... they wind up in the render array and are sort of part of the public API.

We should just be able to use the equivalent one-line fix that was committed to Drupal 8, right? (Plus the tests of course.)

das-peter’s picture

StatusFileSize
new2.09 KB

@David_Rothstein Do you think something like the attached patch?

I hope I got the conditions right ;) The condition should act like the before newly introduced property html - either it's html or not.

bisonbleu’s picture

Ran into this issue. Applied patch in #58. No fuss. Works great. No side effects yet.
Thanks everyone!

Shevchuk’s picture

Status: Needs review » Reviewed & tested by the community

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 58: options_drupal7-1919338-58.patch, failed testing.

Status: Needs work » Needs review
David_Rothstein’s picture

-      if ($properties['strip_tags']) {
-        $options[$value] = strip_tags($label);
-      }
-      if ($properties['filter_xss']) {
+      if ($properties['filter_xss'] || !$properties['strip_tags']) {
         $options[$value] = field_filter_xss($label);
       }
+      else {
+        $options[$value] = decode_entities(strip_tags($label));
+      }

Sorry, I still don't understand why we're changing the logic here? This patch would make it so setting 'filter_xss' and 'strip_tags' both to FALSE still results in field_filter_xss() being called...

I don't see why we'd want that or how it's related to this issue. Why can't we just add decode_entities() as was suggested originally?

+    $this->assertRaw('Some HTML encoded markup with &lt; &amp; &gt;', 'HTML entities in option text were properly handled and not double-encoded');

The addition of the assertion message is a good idea - this should be in Drupal 8 also (can be a separate issue).

dewalt’s picture

Status: Needs review » Closed (works as designed)

I think there is no needs to decode HTML while processing options. '#options' property and hook_options_list() should provide unescaped values, which should be escaped before output. (As works now)

If '#options' ( or hook_options_list() output ) contains the escaped values - it is a bug in module, implements the form element.

bradjones1’s picture

Status: Closed (works as designed) » Needs review

Moving this back to needs review. Many people are finding this issue from the entity reference issue referenced above, and from the discussion it appears there is good consensus that this should be addressed in core, rather than in module implementation. At the very least, I'd be more comfortable with a core maintainer making the call that this is works as designed.

anybody’s picture

I can confirm that the original issue #1665818: Ampersands and other characters are not displayed correctly in <select> widgets still exists and is a big problem. Anyway I think there is a big potential to security issues in both of these issues so that a core developer or even a member of the security assurance team should definitely have a look at it.

Thanks a lot for you work so far! Let's bring this forward!

emmajane’s picture

We are also having problems with this. From the comments it looks like there were a few new changes proposed in #58 which need to be removed to make the patch closer to #14?

guillaumev’s picture

#58 works well on my side, but would feel more comfortable if a core maintainer could confirm that the patch is indeed going to be accepted in core.

das-peter’s picture

@David_Rothstein I was relying on patch #48 when creating #58.
The logic basically is: "You want html? Ok, no strip_tags() for you then, but you shall not pass without field_filter_xss()".
field_filter_xss() allows these tags 'a', 'b', 'big', 'code', 'del', 'em', 'i', 'ins', 'pre', 'q', 'small', 'span', 'strong', 'sub', 'sup', 'tt', 'ol', 'ul', 'li', 'p', 'br', 'img' which seems pretty fair.

I personally am not sure how much I like the ability to disable all the filtering.
However, fact is, that currently it is possible to disable all the filtering.

  • Changing: could be considered as API change
  • Leaving: could be considered as not fixing a security issue.

So who can decide on this?

star-szr’s picture

Issue summary: View changes
Issue tags: -Novice

I updated the proposed resolution section of the issue summary in an attempt to outline the proposed solutions for D7 so far (I count 4). Removing the Novice tag because there is no current Novice task.

I'm not sure why what was committed to D8 (#38) may not be suitable for D7. A next step may be to do a direct backport of #38 and see what testbot thinks of it. We could also look at adding the additional test assertion (see #47/#48).

mpdonadio’s picture

@Cottser, it this what you are talking about? The test-only failed locally, and the full patch passes locally.

The last submitted patch, 71: select_widget_from_the-D7-1919338-test-only.patch, failed testing.

dewalt’s picture

@mpdonadio, the solution in patch will have another case. If I'll define somethin like "select&copy" in options list for widget, I'll not see expecting result, I'll see "select&©".

I think, that current implementation is complete, the solution can be to complete hook_options_list documentation with "The labels should NOT be sanitized and HTML encoded" to eliminate misunderstandings. By the way, HTML entities encode is a one of sanitizition types (http://php.net/manual/en/filter.filters.sanitize.php) - and I'm very suprized that Damien Tournoud in #32 said, that they are different things.

In this way the problem goes to entityreference module level, ( I don't know any other module, the issue takes place ). Basicly, #1665818: Ampersands and other characters are not displayed correctly in <select> widgets has a green patch.

damien tournoud’s picture

@dewalt: If you look at the description of the options list for the widget, it is clearly stated that the labels are HTML:

  $form['allowed_values']['#description'] .= '<p>' . t('Allowed HTML tags in labels: @tags', array('@tags' => _field_filter_xss_display_allowed_tags())) . '</p>';

If you entered select&copy as the label, you entered invalid HTML. If you entered invalid HTML, you will have something garbled in the output. This is really where the story end. If you want correct output, enter correct HTML, which is here select&amp;copy.

dewalt’s picture

@damien-tournoud, it means, that there is list of allowed tags only, but not Full HTML is allowed. There is nothing about HTML entities encode in this description.

The same time, options module API says:

The labels should NOT be sanitized,
options.module takes care of sanitation according to the needs of each
widget. The HTML tags defined in _field_filter_xss_allowed_tags() are
allowed, other tags will be filtered.

The part with HTML tags is clear, and for the other cases (including HTML entities encode) there is should be used the next instruction: " The labels should NOT be sanitized."

Here is full docstring:

/**
 * Returns the list of options to be displayed for a field.
 *
 * Field types willing to enable one or several of the widgets defined in
 * options.module (select, radios/checkboxes, on/off checkbox) need to
 * implement this hook to specify the list of options to display in the
 * widgets.
 *
 * @param $field
 *   The field definition.
 * @param $instance
 *   (optional) The instance definition. The hook might be called without an
 *   $instance parameter in contexts where no specific instance can be targeted.
 *   It is recommended to only use instance level properties to filter out
 *   values from a list defined by field level properties.
 * @param $entity_type
 *   The entity type the field is attached to.
 * @param $entity
 *   The entity object the field is attached to, or NULL if no entity
 *   exists (e.g. in field settings page).
 *
 * @return
 *   The array of options for the field. Array keys are the values to be
 *   stored, and should be of the data type (string, number...) expected by
 *   the first 'column' for the field type. Array values are the labels to
 *   display within the widgets. The labels should NOT be sanitized,
 *   options.module takes care of sanitation according to the needs of each
 *   widget. The HTML tags defined in _field_filter_xss_allowed_tags() are
 *   allowed, other tags will be filtered.
 */
function hook_options_list($field, $instance, $entity_type, $entity) {
damien tournoud’s picture

@dewalt:

it means, that there is list of allowed tags only, but not Full HTML is allowed. There is nothing about HTML entities encode in this description.

It is either plain text, or it is HTML, there is nothing in between. When you are in an HTML context, some character sequences have special meaning, most notably <, > and &.

The part with HTML tags is clear, and for the other cases (including HTML entities encode) there is should be used the next instruction: " The labels should NOT be sanitized."

You are confusing encoding and sanitization. You are not the first one, and most probably you will not be the last. Labels are HTML, which means that <, > and & have special meaning. And *in addition*, only some tags are allowed.

The (arguably confusing) description here just says that (1) labels are HTML and (2) there is no need to sanitize that HTML (i.e. verify that it doesn't contain dangerous tags), because the options module is doing that itself.

dewalt’s picture

StatusFileSize
new11.16 KB

@damien-tournoud, as I see there is no answer in documentation, is label value HTML or not. But I looked wider and now I see the problem in core modules. I compared 'options_select' and 'options_buttons' widgets behaviour in options module.

I've used the same allowed values list:

1|One <b>big</b> & <good> option
2|Second <i>small</i> &copy; text


but got different results

Options widgets display comparation

The question is, what behaviour is right?

mpdonadio’s picture

#77, is that HEAD or w/ patch applied?

dewalt’s picture

#78, I've tested on Drupal 7.32 without patches. Today I'll recheck with HEAD of 7.x.

JvE’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community
StatusFileSize
new4.97 KB
new5.15 KB

Using valid html:

1|One <b>big</b> &amp; &lt;good&gt; option
2|Second <i>small</i> &copy; text

On an unpatched D7.39:

After applying the patch from #71

Can we get this committed?

mpdonadio’s picture

#79, if you do any testing, please do it against the latest version or HEAD, and also again with the patch. Just testing current behavior won't move this patch forward. However, #80 shows before and after screenshots, which I think addresses all concerns with the issue as described.

damien tournoud’s picture

+1, thanks @mpdonadio & @JvE

tistou’s picture

Subscribing.

colan’s picture

@tistou: It is no longer necessary to add comments to subscribe to an issue. If you simply wish to subscribe, please use the Follow button on the top right.

nithinkolekar’s picture

#71 works fine with 7.39 but why it is not committed to 7.39 (released Aug ) though patch was RTBC 2 months back?

David_Rothstein’s picture

Status: Reviewed & tested by the community » Needs review

@nithinkolekar, Drupal 7.39 was a security release so it couldn't have included this patch.

So, looking at this patch more closely, there's no doubt decode_entities() is the correct thing to do, but what concerns me a bit is that we're changing the meaning of the 'strip_tags' option. Before it did exactly what it said, and there are some HTML contexts in which it would have been safe on its own. Now the addition of decode_entities() means you definitely can't output it to HTML without running it back through check_plain() again.

So I think we should consider a more backwards-compatible solution here which is to mark 'strip_tags' as deprecated and introduce a new option that does exactly what we want.

I realize these options are all mainly intended for internal use within the Options module, but we can't know for sure that's actually how it is being used - and when it comes to security it is better to be as safe and as backwards-compatible as possible.

David_Rothstein’s picture

StatusFileSize
new2.9 KB
new1.5 KB

Here is a patch to review that does that.

natanmoraes’s picture

To me, #87 works and looks good

netw3rker’s picture

#87 worked for us too! Thanks @David_Rothstein !

mpdonadio’s picture

anybody’s picture

Status: Needs review » Reviewed & tested by the community

#87 works gread and looks good. So I'd set RTBC for #87 and agree with #90!
Does everyone agree?

Anonymous’s picture

#87 works for me, too!

lachezar.valchev’s picture

Hi,

#87 Works great for me as well.

blainelang’s picture

#87 also worked for me - thanks!

i.bajrai’s picture

#87 works!

valthebald’s picture

Issue tags: -Needs backport to D7

Removed 'Needs backport' tag

stephesk8s’s picture

#87 worked for me. Thank you!

anybody’s picture

Will this be part of the next stable D7 core release?

David_Rothstein’s picture

Status: Reviewed & tested by the community » Fixed
Issue tags: +7.42 release notes

Committed to 7.x - thanks!

  • David_Rothstein committed 50c9b28 on 7.x
    Issue #1919338 by catch, bradjones1, mpdonadio, David_Rothstein, Cottser...
David_Rothstein’s picture

Issue tags: +Needs backport to D7

Also restoring the backport tag for posterity.

benjy’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.