Problem/Motivation
Child of #2506445: Replace !placeholder with @placeholder in t() and format_string() for non-URLs in tests and part of #2506427: [meta] !placeholder causes strings to be escaped and makes the sanitization API harder to understand.
Proposed resolution
Replace as many ! with @ as possible and see what happens. Giving the Views its own issue since there's a lot of !placeholder, it has an important admin UI, and this probably needs to be tested manually as well.
Grep command:
egrep -r '\Wt\(.*\!([a-zA-Z])+' * | grep -v Test.php | grep -v 'href="!' | grep -v "vendor" | grep -v "\.js" | grep -v "\.css" | grep -v "~" | grep "core/modules/views"
Beta phase evaluation
Remaining tasks
TBD
User interface changes
Ideally, fewer completely unexpected double-escaping bugs.
API changes
None.
| Comment | File | Size | Author |
|---|---|---|---|
| #103 | interdiff-97-103.txt | 1.58 KB | stefan.r |
| #103 | 2506479-103.patch | 16.14 KB | stefan.r |
| #98 | 2506479-97.patch | 17.9 KB | stefan.r |
| #98 | interdiff-93-97.txt | 3 KB | stefan.r |
| #93 | 2506479-93.patch | 21.06 KB | stefan.r |
Comments
Comment #2
xjmMissed one.
Comment #4
xjmProbably should also get
ViewEditTestin the same patch, so adding that now.Comment #5
xjmComment #6
xjmI'd forgotten how scary
DisplayPluginBaseis. Keep children and small animals away!Comment #7
xjmNote that this patch only does
t(), notString::format().Comment #9
xjmSo the failure above is because the
<script>tag in the block label is now being double-escaped instead of just single-escaped in the Views block description.Comment #10
xjmSo, this is weird. I tried creating the similar view with block display manually, and then I don't get the double-escaping bug -- using the title copied exactly from the test.
Test verbose output
Testing manually
Meow?
Comment #11
xjmNote that I also tested in Stark and got the same result (single escaping).
Comment #12
dawehnermmmh
Comment #13
lauriiiI can reproduce the double escaping manually. Just did the steps from the test and kaboom!
Didn't have time to figure out this more right now. I might take a look on this later if this haven't been solved by then :)
Comment #14
xjmThis was superseded by #2506445: Replace !placeholder with @placeholder in t() and format_string() for non-URLs in tests, so closing as a duplicate.
Comment #15
xjmOkay, there are a lot of these in Views that bloat the other patches... so resurrecting this, but postponing for now.
Comment #16
xjmI updated the other issue to exclude Views directories, so actually this one can be active again. Scary though. :)
Comment #17
xjmComment #18
xjmComment #19
effulgentsia commentedWould it be helpful or not helpful to break this up into "Views" and "Views UI"?
Comment #20
effulgentsia commentedWhoever ends up getting started on this first, make whatever judgment call you want to make on #19. We can always split or merge the patches later if need be. The important thing is to get started.
Comment #21
effulgentsia commentedPromoting to Critical for the same reason as #2566503-15: [meta] Replace remaining !placeholder for Non-URL HTML outputs only.
Comment #22
lauriiiComment #25
dawehner* Double escaping on admin/structure/views-
-
-
-
-
-
-
-
-
* Views edit page looks great, both with aggregation enabled and disabled
* Escaping of tags on modal titles (might be HEAD)
* Deleting a display / view works fine
* Custom "administrative title"s for fields/filters are double escaped
* display title is getting xss filtered
* Administrative comment is escaped once, and just once
@group is escaped once, yeah, same with title. Perfect
These are machine names, so this is totally fine
Looks perfect
Duplicated code
Follow up: pointless t() call
Don't we loose the safety information again here, especially because $also_known is an array of t() resulsts
General idea, could we disable the serialization of unsafe TranslateWrapper instances?
Does anyone mind opening up a follow up replacing that with $user->getUsername() which is IMHO the right usage here?
Checked that, this works, beside the previous mentioned double escaping
Comment #26
dawehner.
Comment #27
sutharsan commentedYes, we do. This is a fine example of how complex it can get with nested t()s. Below is an excerpt of existing code that is leading up to the
t('Also known as:. We have three levels of nested t()s each with placeholders.Comment #28
dawehnerSo here is a thing, I think all beside the last t() call are actually pointless, IMHO.
Comment #29
sutharsan commentedI always thought that t calls with only placeholders like
t('@label (!name), ...where useful for right to left languages. I checked Arabic and Hebrew, but both don't change the order of the placeholders in this kind of strings. They do change the order when text and placeholders are combined (but not always). I also checked for languages with different character sets (Russian, Japanese, Hindi) to verify if they use ':' and '(' unchanged. And they do. So, in the examples I have found, these placeholder-only strings don't need to be translated.Comment #30
dawehner@Sutharsan
You indeed had the same idea as I had in the ViewsBlock conversion issue. Same problems and finally nobody actually remember why, beside tricking out safemarkup.
Comment #31
catchSo that's true except for at least French, which does:
Something : something else.Instead of
Something: something else.With RTL the actual directionality of the browser should cover something like t('@label (!name)'), but I think we need to cover the colon translatability, or use something other than a colon.
Comment #32
xjmRight -- punctuation, etc. are language-specific. Parentheses also might be represented by other characters or other grammatical constructions in other languages as well (or need additional words to clarify that are OK to omit in English). So we cannot assume that it's safe to remove those
t()if we are combining two strings, possibly themselves also translatable strings, in a way that needs to be comprehensible.Comment #33
dawehnerSo yeah, well the only problem is then
which we could convert to inline templates, which meh, would slow down the cold cache (no idea about the numbers, but this scales with the amount of fields you have in the system).
Comment #34
larowlanworking on fails, assuming this time they are real
is #2207247: Dialog titles double escaped for views handlers and delete forms
Comment #35
larowlanfirst a reroll
Comment #38
larowlanFixed the nits but couldn't fix the fails.
It looks like the second drupalGet in the UITestBase is hitting the parent site instead of the child, the 404 seen is from the parent site. Which explains the 404 - because the view doesn't exist.
If I enable the view on the parent site and add the block and configure it as per the test - the test passes.
Which indicates definitely hitting the parent site. But can't see anything in the patch that would effect that.
Comment #40
effulgentsia commentedFor the implode() part, do we not have a way to implode() safe strings and have the result be safe other than via a inline_template? If not, then yes, this needs an inline template, and perhaps we need a separate issue for having a PHP version of safe_join?
For the
.= ' ' .part, might that not also depend on language? Should we fix that by making the t() broader, such as:Comment #42
effulgentsia commentedAlso, re #40, should we pull out things that require more thought into a separate issue, and only deal with the more straightforward ones here?
Comment #43
dawehner+1 good idea!
Comment #44
plachWorking on test failures...
Comment #45
plachThe test was failing because we were trying to put an HTML-escaped string containing HTML-sensitive characters (
<test>) in an HTML attribute. This is wrong as in that context we should not apply HTML markup sanitization:That obviously cannot work and makes text assertions fail. I think we need to address #2509218: Ensure that SafeString objects can be used in non-HTML contexts first to fix those, the attached patch reverts the related changes for now.
Comment #48
plachJust in case anyone were wondering: unescaped
<and>are legal characters inside attributes:Comment #49
wim leers#48: oh wow!
Comment #50
plachSo, the following source is legal, according to https://validator.w3.org/#validate_by_input:
However, as correctly pointed out by @stefan_r, XML will break. And the following is needed to make the validator happy:
The code above works as intended (even in its HTML5 flavor) so I'd say I was wrong and we can just escape
<and>and change the test assertions. Trying that.Comment #51
plachNope, code is actually double-escaped so we need #2509218: Ensure that SafeString objects can be used in non-HTML contexts.
Comment #52
effulgentsia commentedLet's limit this issue to non-attribute-value output then. Retitling.
Comment #53
dawehnerCool, its progress!
Comment #54
effulgentsia commented@catch reminded me this morning that the preferred way to do this is:
Unless there's reasons why this is a bad idea for this particular part of Views (e.g., if the output is for an attribute value, or performance concerns about full rendering in this part of the pipeline).
Note that as a separate issue, I think we should consider allowing render arrays themselves as values of '@' placeholders, instead of requiring the caller to early render.
Comment #55
effulgentsia commentedSorry. Unintentional title change in #54.
Comment #56
dawehner@effulgentsia
We do have a couple of inline templates in core, should we then also use a proper render array instead?
Comment #57
effulgentsia commentedAre we sure this isn't creating double escaping? If it's not, why isn't it?
Comment #58
dawehnerIt will probably indeed, see some of my earlier reviews. We though should also add some tests, just to be sure.
Comment #59
effulgentsia commentedI guess we'd need separate issues to evaluate each one case by case. template_preprocess_search_result() and Field::renderItems() seem like possible candidates, though they don't use "," as the separator, so maybe that's why they don't use "comma-list"?
Comment #60
joelpittetI'm not entirely sure where this is rendered out. Was assuming Content ID = also known as nid in the views UI would be it but that looks broken in head and this patch...
I could be wrong about the location of this Also known as concatenation. We can add double escape assertion if we can find where this alias help is rendered.
Comment #61
dawehnerYeah, it though would be on a configurable field, like the node body.
Comment #62
dawehnerSteps to manually reproduce that:
Rename the label of the body field on articles to "Description". Once done the following thing appears:

Given that we talk about field labels, I don't think we will ever ever have HTML there
Comment #63
catchEven if someone renames a boolean field to
Toggle <em> tagthen the em tag should be treated as plain text and escaped, so I think that's fine.Haven't reviewed all the rest of the patch yet so just leaving RTBC for now.
Comment #64
alexpott$also_know is an array of already escaped string - this is ripe for double escaping.
What is happening here?
Comment #65
dawehnerAs said earlier, the content will never have HTML.
Comment #66
catchIs there a follow-up for the t() return passed into Attributes that was split out of this patch? I can't find one mentioned anywhere, or in the queue.
Comment #67
stefan.r commentedWhat about quotes and ampersands?
Comment #68
dawehnerWell, do you use them for field labels?
Comment #69
lauriiiTried to make some progress.. Maybe not the nicest solution ever
Comment #70
dawehnerHa, we have no tests at all for this functionality, this is for sure.
You run into the following error message with your approach:
This here is another classical place where you ideally want to construct just a string, because at that point in time its not yet meant for generating HTML. The level of outputting will be done much latter, so I actually think we should avoid to escape it here. We construct the string and then later could just deal with it.
Comment #71
lauriiiAdded some docs to the descape method. Thanks @dawehner for the added test coverage.
Comment #72
stefan.r commentedIt's a bit sad that we need this but it'll solve our label conundrum :)
unneeded?
Might have been, or have been? The t() string has @placeholders, so unless those are safe (I don't think they are here), they'll be escaped.
We have the comma list template for this, can't we use that?
Let's mention in the long description how it's the inverse of self::escape()?
Should we also mention use cases along with a warning not to misuse this?
Let's add a unit test in HtmlTest
Comment #73
lauriii#72.3: It might change at some point and then we would have to change the comment
#72.4: It used to be implode and because of the fact that is concatenated into a string there is no real benefit to use the template
Fixed rest of #72.
Comment #74
lauriiiHmm, files lost again?
Comment #75
stefan.r commentedI think this is RTBC, asked @kgoel to have a final look at this and document in HtmlTest how they're inverse functions by doing some
Html::descape(Html::escape((x))) == xComment #76
lauriiiComment #77
star-szrDid we consider naming this method unescape instead? curl and JavaScript either have or used to have unescape (see below) and descape seems to be our own invention.
http://curl.haxx.se/libcurl/c/curl_easy_unescape.html
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global...
Minor: Missing period on the second line.
Minor: Attach the same what? In other words, I think we should say "same field" here.
Probably minor: Why is the second label not translated? Might relate to #25.5.
Comment #78
cpj commentedComment #79
cpj commentedComment #80
joelpittet+1 for unescape instead of descape
This t() should be removed, we are writing an assert that makes this unpossible:)
This seems like a bunch of processing, (translate in __toString on TranslationWrapper, implode, de-escape. But worse is the : feels like it's missing context in the 'Also known as: t()'. Can we put that back in t('Also known as: @also')?
Comment #81
lauriiiComment #82
lauriiiFixed all the things except #80.2 which requires quite a few changes to make work without double escaping problems
Comment #89
joelpittetThank you for cleaning up the other issues. @lauriii #80.2 is a bit of a hard problem.
I do like the function balancing out Html::escape() but I wonder if this issue needs to be the one to add it.
Maybe this help should be built up some other way?
Can we get @lewisnyman to have a quick look at this CSS? May need a class instead of targeting the span tag.
Should operators be translated, would they change since they are programming terms? They seem to me not to be translated because they won't change between language.
Should we use a render array here to concatenate or inline template?
Comment #90
dawehnerIt feels like views is not a programming UI, but its something for sitebuilders. I think keeping the translatability is crucial.
Comment #91
catchI couldn't find the follow-up so added a stub #2571673: Convert Views t() usage where it is used as an attribute value.
Comment #92
dawehnerThat looks alright! Even you might think, this is not really underspecified CSS.
Comment #93
stefan.r commentedAddressing #89.1
We had the same problem in #2296929: Remove system_requirements() SafeMarkup::set() use and #2550981: Remove SafeMarkup::set in FormErrorHandler::displayErrorMessages(). Regardless of whether the patterns used there were the best possible fix, maybe instead of introducing yet another pattern let's just go with the same pattern here? It's a bit sad that we can't do t('something: @further_context') anymore, but we didn't do that in the linked issue either.
The unescape looks nice, but as it's not needed if we use markup arrays and the comma list template, that could go into its own issue if we want to do this instead.
Comment #95
dawehnerI think its fundamental wrong what we do here. We don't want to generate HTML here.
Comment #96
stefan.r commented@dawehner well then the problem is we use t() in a plain-text context, as t() is currently designed to work with markup. Which brings us back to #2509218: Ensure that SafeString objects can be used in non-HTML contexts.
Let's discuss with @alexpott & @xjm when we get to the CCIB
Comment #97
wim leersI found no additional remarks beyond #95. I think this is ready once it's green.
Just discussed this with Alex Pott, he's +1 to splitting off #95 into a separate issue. Created that issue: #2571935: Fix use of !placeholder for imploding in views.views.inc.
Comment #98
stefan.r commentedComment #100
dawehnerShould we just add a test so we know exactly how the "expected" behaviour is?
Comment #101
stefan.r commented@dawehner yes, maybe in #2571935: Fix use of !placeholder for imploding in views.views.inc as I don't think the current patch touches that part anymore?
Comment #102
dawehnerYeah good point. Let's get this in, once it is green!
Comment #103
stefan.r commentedComment #104
alexpottComment #105
dawehnerThanks drupalCI!
Comment #106
alexpottCommitted 3f689d7 and pushed to 8.0.x. Thanks!
Comment #110
mariancalinro commentedChanging to fixed, as it's already committed, and the test failed because the patch couldn't be applied.
Comment #111
hass commentedThis patch contains a lot of bugs.
This is incorrect code. This is correct way of doing this:
Same bug
Same again
Same again
Same again
Same again
Comment #112
dawehner@hass
You are right about the fact that we do it wrong in views, but this issue about having a straight conversion to fix XSS issues, not about fixing views.
Do you mind open up a new issue?
Comment #113
hass commented