Problem/Motivation
This is coming from #1665818: Ampersands and other characters are not displayed correctly in <select> widgets in the Entity reference queue but the bug reported there can be found in core now that ER is in core.
When implementing hook_options_list() and passing the options through check_plain(), any options with ampersands or < and > symbols end up double encoded in select widgets.
Quoting Damien Tournoud in #1665818-19: Ampersands and other characters are not displayed correctly in <select> widgets:
Core does a
strip_tags()without adecode_entities()
Proposed resolution(s)
Drupal 7:
A) #38 (no D7 patch yet, this is what was committed to D8)
B) #18 (no patch yet, differs from what was committed to D8)
C) #48 (sent back in #57)
D) #58 (sent back in #63)
Drupal 8:
Add decode_entities() to the strip_tags block in _options_prepare_options().
Remaining tasks
- Needs tests.
- Update taxonomy module so the term reference field does a
check_plain()in itshook_options_list()implementation?
User interface changes
n/a
API changes
TBD
Potentially related issues
| Comment | File | Size | Author |
|---|---|---|---|
| #87 | interdiff.txt | 1.5 KB | David_Rothstein |
| #87 | select-widget-options-D7-1919338-87.patch | 2.9 KB | David_Rothstein |
| #77 | options_widgets.png | 11.16 KB | dewalt |
| #71 | select_widget_from_the-D7-1919338-71.patch | 1.91 KB | mpdonadio |
| #71 | select_widget_from_the-D7-1919338-test-only.patch | 1.33 KB | mpdonadio |
Comments
Comment #0.0
star-szrTweaking remaining tasks formatting
Comment #1
star-szrComment #3
damien tournoud commentedThis whole function needs to be refactored, because there are only two possible cases, not four. A simple
$properties['html'](true/false) would be enough.Also, I usually recommend
trim(str_replace("\n", " ", drupal_html_to_text($value)))as a way to down-case HTML to plain-text, because it results in a plain-text string that translates more of the intention of the HTML (for example,<em>text</em>is going to be transformed into*text*).Comment #4
star-szr#1: 1919338-1.patch queued for re-testing.
Comment #5
JvE commentedWould it not be easier to remove the check_plain()s from form_select_options() rather than
?
Comment #6
damien tournoud commented@JvE: no, it would not. Select elements require plain-text.
Comment #7
JvE commentedThe current encoding flow for taxonomy select is:
The current encoding flow for entityreference select is:
Leading to double encoding.
This patch changes that to:
and suggests changing taxonomy select to do the same.
Comment #8
damien tournoud commented@JvE: yes, what you describe is the intended flow.
hook_options_list()specifies that its values are HTML:In most cases, we already have HTML. For example, the allowed values for the field types managed by the list module are HTML.
But both Entity Reference (when using the base selection plugin) and Taxonomy Reference only have a plain-text label, so they need to transform it to HTML by using
check_plain().Inside the options module, we have two cases:
filter_xss()for sanitization#optionsfor the select widgetThis last process is where the bug is. The current code only does a
strip_tags()without adecode_entities(), which result in broken output. I recommend we change this totrim(str_replace("\n", " ", drupal_html_to_text($value)))so as to generate a plain-text value that keeps some of the intend of the original HTML.Comment #9
sunThere's a third option, which is to finally get rid of the inconsistency of #options with regard to #type checkboxes, radios, and select:
#342316: Introduce proper Form API #types for 'option' and 'optgroup', and make #options consistent.
Comment #10
damien tournoud commented@sun, yes, that makes sense, but that's not backportable. Let's come up with a solution we can actually use in 7.x.
Comment #11
star-szrThis probably needs be reconsidered now that #1751234: Convert option widgets to Plugin system is in. Is this fixed in 8.x now, and if so can we move this to 7.x?
Comment #12
catchThat patch doesn't change anything functionally, here's the relevant hunk:
Comment #13
catchHere's a patch, test-only, then test with fix for Drupal 8.
Comment #14
catchAdding Drupal 7 backport, moving version temporarily for the bot.
Comment #15
JvE commentedYes, I like it.
Comment #16
catchBack to 8.x again.
Comment #17
star-szrD8 patch in #13 looks good to me and has tests. Unfortunately it needs a reroll now.
Comment #18
damien tournoud commentedIn #8, I suggested using
trim(str_replace("\n", " ", drupal_html_to_text($value)))instead, because it keeps more of the semantics of the original HTML. Opinions on this?Comment #19
rob230 commentedI've used #14 on my D7 site. It fixes the double encoding and I've not had any problems so far.
Comment #20
catch@Damien the strip_tags() is already there, this just fixes the double-encoding bug so I don't see any reason to change the intended behaviour here, might be worth opening another issue though.
Comment #21
shnark commentedReroll.
I followed these instructions. https://drupal.org/patch/reroll.
There were no conflicts that I had to solve by hand:
$ git rebase 8.x
First, rewinding head to replay your work on top of it...
Applying: applying patch
Using index info to reconstruct a base tree...
M core/modules/options/lib/Drupal/options/Plugin/field/widget/SelectWidget.php
M core/modules/options/lib/Drupal/options/Tests/OptionsWidgetsTest.php
Falling back to patching base and 3-way merge...
Auto-merging core/modules/options/lib/Drupal/options/Tests/OptionsWidgetsTest.php
Auto-merging core/modules/options/lib/Drupal/options/Plugin/field/widget/SelectWidget.php
I looked at the patch from comment #13. The changes in that are similar looking to the changes in the reroll.
Comment #22
shnark commentedComment #23
shnark commentedComment #24
star-szr#21: 1919338_21.patch queued for re-testing.
Comment #25
ptvv commentedSurely one of the big problems here is precisely
hook_options_list()'s documentation. If you include the previous sentence too the quote isIf the return values are supposed to be HTML, then they should be HTML-encoded already and the first sentence is incorrect: the only sanitisation that should be necessary is anti-XSS filtering, which should be idempotent, so rather than saying that the labels should "NOT" be sanitised it should say that it is not necessary to perform anti-XSS filtering on them.
On the other hand, if the return values are supposed to be plain text then no HTML tags should be allowed.
Either way, the documentation is at best misleading and at worst is actively encouraging developers to create bugs.
Comment #26
mustanggb commentedBeen running the D7 port without any problems.
Comment #26.0
mustanggb commentedRemove unneeded comment number
Comment #27
wiifmAfter reading the whole of #1665818: Ampersands and other characters are not displayed correctly in <select> widgets and then finding this issue, I applied @catch's patch at #14 and now my ER select list is no longer double encoded (Drupal 7).
+1 for that.
Comment #28
jamesdixon commentedD7 patch #14 worked great for me on Drupal 7.23.
I had entity reference fields with select widgets showing html entities incorrectly, but patch #14 cleared that up.
Thanks @catch and everyone who contributed to the fix.
Comment #29
bblake commentedRolled against 7.22
Comment #31
star-szr#21 is the patch to review for D8.
Comment #32
damien tournoud commentedThat's just because you confuse "encoding" (which the action of transforming plain-text into HTML), with "sanitization" (which is the action of making sure that the HTML doesn't contain executable code). The documentation is misleading, but it is technically correct.
Comment #33
jwilson3Patch in #14 also solves the problem visible in Drupal 7 Organic Groups select list on node edit forms for og-enabled content types, when an ampersand appears in a Group name.
Comment #34
webadpro commentedPath #14 did the trick for me also. When will this be pushed in the core for D7?
Comment #35
dooleysarahe commented+1 patch #14 for D7. It would be great to see this committed to core.
Comment #36
alauddin commented#14 patch for D7.26 as well.
Comment #37
mustanggb commentedTagging novice for a re-roll.
Comment #38
swentel commentedrerolled
Comment #39
andypostfinally, let's get this in! #13 has "fail-test"
Comment #40
alexpottCommitted 966c330 and pushed to 8.x. Thanks!
Comment #41
bradjones1Rerolled #14 for D7.
Comment #44
bradjones1Let's try this again.
Comment #46
bradjones1No longer needs tests. I'm going to RTBC as I'm not the original author of the patch.
Comment #47
jwilson3It wouldn't hurt to write an assertion message for the second test, to make it clear what you're testing for. Something like 'HTML entities in option text were properly handled and not double-encoded'.
Comment #48
Jalandhar commentedUpdating the patch as said in comment #47, Please review.
Comment #49
wiifmPatch looks good to me, and fixes a fairly major bug with Drupal core at present. +1 RTBC
Comment #51
Jalandhar commented48: options_drupal7-1919338-48.patch queued for re-testing.
Comment #52
Jalandhar commentedI was not sure why the test had failed earlier but it got passed now after re-testing...:P
Comment #53
wiifmBack to RTBC due to the testbot having a bad day.
Comment #55
Jalandhar commented48: options_drupal7-1919338-48.patch queued for re-testing.
Comment #56
Jalandhar commentedBack to RTBC.
Comment #57
David_Rothstein commentedI don't think we should change all those property names or merge them... they wind up in the render array and are sort of part of the public API.
We should just be able to use the equivalent one-line fix that was committed to Drupal 8, right? (Plus the tests of course.)
Comment #58
das-peter commented@David_Rothstein Do you think something like the attached patch?
I hope I got the conditions right ;) The condition should act like the before newly introduced property
html- either it's html or not.Comment #59
bisonbleu commentedRan into this issue. Applied patch in #58. No fuss. Works great. No side effects yet.
Thanks everyone!
Comment #60
Shevchuk commentedComment #63
David_Rothstein commentedSorry, I still don't understand why we're changing the logic here? This patch would make it so setting 'filter_xss' and 'strip_tags' both to FALSE still results in field_filter_xss() being called...
I don't see why we'd want that or how it's related to this issue. Why can't we just add decode_entities() as was suggested originally?
The addition of the assertion message is a good idea - this should be in Drupal 8 also (can be a separate issue).
Comment #64
dewalt commentedI think there is no needs to decode HTML while processing options. '#options' property and hook_options_list() should provide unescaped values, which should be escaped before output. (As works now)
If '#options' ( or hook_options_list() output ) contains the escaped values - it is a bug in module, implements the form element.
Comment #65
bradjones1Moving this back to needs review. Many people are finding this issue from the entity reference issue referenced above, and from the discussion it appears there is good consensus that this should be addressed in core, rather than in module implementation. At the very least, I'd be more comfortable with a core maintainer making the call that this is works as designed.
Comment #66
anybodyI can confirm that the original issue #1665818: Ampersands and other characters are not displayed correctly in <select> widgets still exists and is a big problem. Anyway I think there is a big potential to security issues in both of these issues so that a core developer or even a member of the security assurance team should definitely have a look at it.
Thanks a lot for you work so far! Let's bring this forward!
Comment #67
emmajane commentedWe are also having problems with this. From the comments it looks like there were a few new changes proposed in #58 which need to be removed to make the patch closer to #14?
Comment #68
guillaumev commented#58 works well on my side, but would feel more comfortable if a core maintainer could confirm that the patch is indeed going to be accepted in core.
Comment #69
das-peter commented@David_Rothstein I was relying on patch #48 when creating #58.
The logic basically is: "You want html? Ok, no
strip_tags()for you then, but you shall not pass withoutfield_filter_xss()".field_filter_xss()allows these tags'a', 'b', 'big', 'code', 'del', 'em', 'i', 'ins', 'pre', 'q', 'small', 'span', 'strong', 'sub', 'sup', 'tt', 'ol', 'ul', 'li', 'p', 'br', 'img'which seems pretty fair.I personally am not sure how much I like the ability to disable all the filtering.
However, fact is, that currently it is possible to disable all the filtering.
So who can decide on this?
Comment #70
star-szrI updated the proposed resolution section of the issue summary in an attempt to outline the proposed solutions for D7 so far (I count 4). Removing the Novice tag because there is no current Novice task.
I'm not sure why what was committed to D8 (#38) may not be suitable for D7. A next step may be to do a direct backport of #38 and see what testbot thinks of it. We could also look at adding the additional test assertion (see #47/#48).
Comment #71
mpdonadio@Cottser, it this what you are talking about? The test-only failed locally, and the full patch passes locally.
Comment #73
dewalt commented@mpdonadio, the solution in patch will have another case. If I'll define somethin like "select©" in options list for widget, I'll not see expecting result, I'll see "select&©".
I think, that current implementation is complete, the solution can be to complete hook_options_list documentation with "The labels should NOT be sanitized and HTML encoded" to eliminate misunderstandings. By the way, HTML entities encode is a one of sanitizition types (http://php.net/manual/en/filter.filters.sanitize.php) - and I'm very suprized that Damien Tournoud in #32 said, that they are different things.
In this way the problem goes to entityreference module level, ( I don't know any other module, the issue takes place ). Basicly, #1665818: Ampersands and other characters are not displayed correctly in <select> widgets has a green patch.
Comment #74
damien tournoud commented@dewalt: If you look at the description of the options list for the widget, it is clearly stated that the labels are HTML:
If you entered
select©as the label, you entered invalid HTML. If you entered invalid HTML, you will have something garbled in the output. This is really where the story end. If you want correct output, enter correct HTML, which is hereselect&copy.Comment #75
dewalt commented@damien-tournoud, it means, that there is list of allowed tags only, but not Full HTML is allowed. There is nothing about HTML entities encode in this description.
The same time, options module API says:
The part with HTML tags is clear, and for the other cases (including HTML entities encode) there is should be used the next instruction: " The labels should NOT be sanitized."
Here is full docstring:
Comment #76
damien tournoud commented@dewalt:
It is either plain text, or it is HTML, there is nothing in between. When you are in an HTML context, some character sequences have special meaning, most notably
<,>and&.You are confusing encoding and sanitization. You are not the first one, and most probably you will not be the last. Labels are HTML, which means that
<,>and&have special meaning. And *in addition*, only some tags are allowed.The (arguably confusing) description here just says that (1) labels are HTML and (2) there is no need to sanitize that HTML (i.e. verify that it doesn't contain dangerous tags), because the options module is doing that itself.
Comment #77
dewalt commented@damien-tournoud, as I see there is no answer in documentation, is label value HTML or not. But I looked wider and now I see the problem in core modules. I compared 'options_select' and 'options_buttons' widgets behaviour in options module.
I've used the same allowed values list:
but got different results
The question is, what behaviour is right?
Comment #78
mpdonadio#77, is that HEAD or w/ patch applied?
Comment #79
dewalt commented#78, I've tested on Drupal 7.32 without patches. Today I'll recheck with HEAD of 7.x.
Comment #80
JvE commentedUsing valid html:
On an unpatched D7.39:
After applying the patch from #71
Can we get this committed?
Comment #81
mpdonadio#79, if you do any testing, please do it against the latest version or HEAD, and also again with the patch. Just testing current behavior won't move this patch forward. However, #80 shows before and after screenshots, which I think addresses all concerns with the issue as described.
Comment #82
damien tournoud commented+1, thanks @mpdonadio & @JvE
Comment #83
tistou commentedSubscribing.
Comment #84
colan@tistou: It is no longer necessary to add comments to subscribe to an issue. If you simply wish to subscribe, please use the Follow button on the top right.
Comment #85
nithinkolekar commented#71 works fine with 7.39 but why it is not committed to 7.39 (released Aug ) though patch was RTBC 2 months back?
Comment #86
David_Rothstein commented@nithinkolekar, Drupal 7.39 was a security release so it couldn't have included this patch.
So, looking at this patch more closely, there's no doubt decode_entities() is the correct thing to do, but what concerns me a bit is that we're changing the meaning of the 'strip_tags' option. Before it did exactly what it said, and there are some HTML contexts in which it would have been safe on its own. Now the addition of decode_entities() means you definitely can't output it to HTML without running it back through check_plain() again.
So I think we should consider a more backwards-compatible solution here which is to mark 'strip_tags' as deprecated and introduce a new option that does exactly what we want.
I realize these options are all mainly intended for internal use within the Options module, but we can't know for sure that's actually how it is being used - and when it comes to security it is better to be as safe and as backwards-compatible as possible.
Comment #87
David_Rothstein commentedHere is a patch to review that does that.
Comment #88
natanmoraesTo me, #87 works and looks good
Comment #89
netw3rker commented#87 worked for us too! Thanks @David_Rothstein !
Comment #90
mpdonadioI'm fine with this approach, but it does mean we will need to adjust dependent issues, like #1793236: views_handler_filter::prepare_filter_select_options() incorrectly converts HTML to plaintext and #1665818: Ampersands and other characters are not displayed correctly in <select> widgets to account for this.
Comment #91
anybody#87 works gread and looks good. So I'd set RTBC for #87 and agree with #90!
Does everyone agree?
Comment #92
Anonymous (not verified) commented#87 works for me, too!
Comment #93
lachezar.valchev commentedHi,
#87 Works great for me as well.
Comment #94
blainelang commented#87 also worked for me - thanks!
Comment #95
i.bajrai commented#87 works!
Comment #96
valthebaldRemoved 'Needs backport' tag
Comment #97
stephesk8s commented#87 worked for me. Thank you!
Comment #98
anybodyWill this be part of the next stable D7 core release?
Comment #99
David_Rothstein commentedCommitted to 7.x - thanks!
Comment #101
David_Rothstein commentedAlso restoring the backport tag for posterity.
Comment #102
benjy commentedThis likely caused #2662432: Undefined index: strip_tags_and_unescape in _options_prepare_options() after updating to 7.42 although not confirmed.