Problem/Motivation
The patch in #2273923: Remove html => TRUE option from l() and link generator. adds the following code:
$log_text = SafeMarkup::set(Unicode::truncate(Xss::filter($message, array()), 56, TRUE, TRUE));
As pointed out in comment #2, the assumption that this is safe because truncating a safe string results in a safe string is not true as truncating a string may lead to a malformed HTML entity.
Proposed resolution
Several approaches for a resolution have been proposed. To summarize these:
- In #3 and #5, a parameter was added to Xss::filter() to indicate whether SafeMarkup should be called before returning the truncated string.
We're not taking this approach because, as stated in #6, adding a parameter to Xss::filter() is out of scope of this issue.
- The next approach (#9, #14, #20, #23, #24, #32) was to add a method to the SafeMarkup class instead of adding a parameter to Xss::filter().
We're not taking this approach because, as stated in #33, it mixes sanitization with SafeMarkup.
- Then a completely new approach (#35, #36, #38, #42, #43, #51, #54, #56, #61) was proposed which would add a method to the Xss class.
We're not taking this approach because, from #62, truncation has nothing to do with Xss filtering and we shouldn't go over the strings twice.
- A minimal patch was provided in #69 which removes does not preserve any HTML tags in the truncated output.
Comments #71 and #72 indicate that this approach is appropriate for this issue as it simply removes SafeMarkup::set().
| Comment | File | Size | Author |
|---|---|---|---|
| #88 | 2399261-88.patch | 4.91 KB | stefan.r |
| #88 | interdiff-85-88.txt | 1.16 KB | stefan.r |
| #85 | 2399261-85.patch | 4.82 KB | stefan.r |
| #85 | interdiff-83-85.txt | 735 bytes | stefan.r |
| #83 | 2399261-83.patch | 4.8 KB | stefan.r |
Comments
Comment #1
star-szrComment #2
pwolanin commentedIt is certainly NOT safe to assume a truncated safe string is safe:
Comment #3
seantwalshTruncating a string may lead to a malformed HTML entity, so we then need to filter it again to make sure all HTML entities are well formed. This removes SafeMarkup::set for the original string, which may be arbitrarily long and only sets as safe the truncated string.
@xjm and @pwolanin discussed this API change in IRC earlier today, which adds an optional, backwards compatible final param $set_safe_markup to Xss::filter().
Comment #4
mpdonadioShould this mention that it is optional and defaults to TRUE?, Maybe
Not sure where 80 cols is there.
Comment #5
seantwalshThanks @mpdonadio, I've changed the text as follows.
Edit: Also updated the issue title.
Comment #6
effulgentsia commented+1 to changing the SafeMarkup::set() to Xss:filter(), but the rest of this patch seems out of scope to me. If we want to add a new parameter to Xss::filter(), can we do that in a separate issue to discuss the motivation for that? Or is there a compelling need to pass in FALSE in the above line?
Comment #7
effulgentsia commentedAlso, why do we need to Xss::filter() the original message to begin with? Why not truncate the unfiltered message, then filter just the once? That would also keep the long original out of the SafeMarkup registry, but without needing a new parameter in Xss::filter().
Comment #8
alexpottFlags are a bit of an anti-pattern http://martinfowler.com/bliki/FlagArgument.html. Why can't we just add a new method for this?
Comment #9
alexpottHere's what I'm suggesting. It has the advantage of one less conditional and being extremely explicit when you are and when you are not making something as safe when using the method.
Comment #10
pwolanin commentedThat's fine for me, as long as we have some way to filter without adding to the safe strings list.
Comment #11
pwolanin commented@effulgentsia the inner call serves to strip all the HTML tags. we want to truncate after stripping the tags so we actually have some meaningful text.
As above, the outer call makes sure we have well-formed html entities after truncating and marks the string safe.
Comment #12
effulgentsia commentedOh, I missed that we were passing an empty tag array in. In that case, why not use strip_tags() instead?
Comment #13
effulgentsia commentedAnd, if we're stripping all tags anyway, can't we then just pass the truncated (not marked safe) message to l() and let it autoescape it? See patch.
Comment #14
effulgentsia commentedHelps to upload one, I guess :)
Comment #15
pwolanin commented@effulgentsia no, that's not right, since it would double escape HTML entities.
Comment #16
pwolanin commentedAlso, strip_tags() should never be used. it's a broken, unsafe function.
Please commit #9.
Comment #17
pwolanin commentedFrom chat w/ effulgentsia , if the tests don't fail on his, that means we are not testing for the presence of a valid HTML entity in the message, so we should add that also.
Comment #19
pwolanin commentedTest fails do look html entity escaping-related:
Comment #20
effulgentsia commentedYay for test coverage!
How about this then?
Comment #21
effulgentsia commentedIf at some point we want to preserve HTML tags in the output, see also #221257: text_summary() should output valid HTML and Unicode text.
Comment #22
pwolanin commentedI still think we do not want to use strip_tags(). We also need a method (or flag) to be able to Xss filter without setting the string into the safe list.
Comment #23
pwolanin commentedHere's the combo patch
Comment #24
effulgentsia commentedWhy not? What's the difference between Xss::filter() with an empty tag list and strip_tags()?
Why add that to this issue unless there's a use case in this issue for it? That goes back then to the prior question.
Furthermore, can we remove the final Xss:filter() by decoding entities before truncation? In fact, wouldn't that improve the truncation, since it would see the entity as a single character? See patch.
Comment #25
pwolanin commentedstrip_tags() is known to be broken in a variety of ways and should not be used: e.g. see http://stackoverflow.com/questions/1972442/when-strip-tags-burns-a-haystack
the Xss::filter() code does work to validate the HTML and entities.
Comment #26
effulgentsia commentedWe use strip_tags() in >300 places throughout core. That stack overflow page only mentions that it strips non-tag text (which is maybe undesirable, but not a security problem) if you give it malformed HTML as input. But in this patch's usage, we're giving it input we made when logging the message, so we have control over it being well formed.
While Xss::filter() works, it creates confusion for why it's being called, because we're not trying to filter for XSS here, we're trying to truncate a string that contains HTML tags and entities in a way that strips the tags and preserves the entities. Using the Xss class for that is overkill and relies on its side effects rather than its purpose.
Comment #27
pwolanin commentedOk, given that we make it plain at the end, maybe the downsides of using strip_tags() are outweighed by the simplicity.
Comment #28
pwolanin commentedActually - think this will still be a bit odd - we will be truncating to the requested number of *visible* characters, but the string could be longer than the requested length because we escape some HTML entities again.
Comment #29
pwolanin commentedIn brief - assume the string to be filtered has
&in it.We would decode it to
&and truncate the string to the desired length, but the final checkPlain() would encode it as&again so now the string is 4 characters longer than requested (though will display in the browser as the requested length).Maybe this is ok if we care about display vs. actual character count?
Comment #30
star-szrI think the behaviour is fine, the docs could perhaps be clearer, the interpretation of "text characters" is a bit up in the air to me:
Comment #31
mpdonadioSince we sanitize for output, and shouldn't be using these eg before shoving strings into the database, perhaps "Truncates an HTML string safely to a number of rendered text characters." ?
Comment #32
effulgentsia commentedClarified docs. Anyone willing to write some unit tests for SafeMarkup::truncate()?
Comment #33
xjmSo TBH I don't think we should be adding more methods with sanitization and HTML manipulation logic to
SafeMarkup; its primary purpose should just be as a value object of safe strings. I think this is perhaps more the domain of the Xss object or maybe the Html object. I checked with @alexpott and he felt similarly.#2506195: Remove SafeMarkup::set() from Xss::filter() already decouples
Xss::filter()along those lines, and also provides a way to avoid the double string list entry we have in HEAD.Comment #34
pwolanin commented@xjm - so should we just inline the various calls here instead of creating a helper method?
It feels like in general people won't necessarily get a safe truncation?
Comment #35
chx commented> It feels like in general people won't necessarily get a safe truncation?
Sure they will if we use the patch attached :) There's no interdiff as this is a brand new approach.
Comment #36
chx commentedSimplified a bit.
Comment #38
chx commentedOh doh, that's just a typo in comments.
Comment #41
joelpittetI really like the direction and idea behind #38 +1 to that! Just need to figure out what's up with the tests.
Comment #42
chx commentedIn order to move this along I am stripping tags from the link as HEAD does. It's not that this code doesn't work -- it's assertLink that throws in the towel when the link text contains HTML. I am working on that http://stackoverflow.com/questions/31419933/xpath-find-link-containing-h... here.
Comment #43
chx commentedDisregard #42, interdiff against #38 this hopefully passes. The only changes are in tests because we now allow <em> in our dblog links.
Comment #44
chx commentedComment #47
joelpittetThat makes those tests better! It leaves
truncateinXssdomain and wraps it inSafeMarkupand less capture meaningless groups captured.Big +1 thanks @chx
Comment #48
joelpittetOh and of course the main reason: it makes the combo
Unicode::truncate(Xss::filterless prone to error and potential security hole vector.Comment #49
alexpottDrupal\Tests\Component\Utility\XssTest.Does not conform to our docs standards.
I think it is more useful to duplicate the documentation here.
Might this not result in a string that exceeds the length?
Comment #50
joelpittet@alexpott re #49.4. Yes if you count tags in the string length, which I would expect we wouldn't be, but I guess either way we need #49.1 to show that our expectations.
Edit: disregard this, @chx went through it with me on IRC.
Comment #51
chx commentedComment #52
chx commentedComment #53
chx commentedHrm my comment got lost: so this still needs tests, the above version is mostly doxygen fix and a few more minor things, like making the db log test assertion even better.
Re .4 the new function does not count the length of HTML tags (open or close) when calculating the length of string. Indeed that's why it's in Xss -- it reuses much of filter but where filter replaces the tags with a safe version and keeps the text intact, this one adds text until it's the desired length then stops.
Comment #54
chx commentedEven more doxygen and a lot of tests. The change the existing filter tests is absolutely minimal and doesn't affect the thing tested (attributes) at all.
Comment #55
alexpottI'm not entirely comfortable with making this change as the logic for determining when an attribute and tag is ending is being tested here.
Comment #56
chx commentedSure, I changed the test itself to expect a slash.
Comment #57
alexpottWorks for me.
Comment #60
joelpittetNice! There is a little indent that could be fixed on commit but I'm for RTBC here.
Bring back 2 spaces on commit please:)
Comment #61
chx commentedWe do not want the committers to overwork themselves so here's two spaces removed.
Comment #62
alexpottI been sleeping on this patch for a while and I think we're entangling two things here - truncating html and filtering. Truncation really has nothing to do with Xss filtering. But in the usage converted in the patch doing truncation and filtering would result in having to loop twice over the html. Anyhow Xss::truncate is mis-named it should be Xss::filterTruncate and similarly for SafeMarkup::xssTruncate - it should be SafeMarkup::xssFilterTruncate... these method names hint at the problem we have. I'm really unsure of what to do about this. Going to ping @xjm and @effulgentsia to see if they have any ideas.
We're missing test coverage of these arguments.
Comment #63
xjmI don't think we should be adding this API at all, especially not now after the work we've done to minimize the API surface.
Is there a reason not to just truncate first and filter after? This patch has gotten a lot more extensive than simply fixing one line in one unimportant controller.
Comment #64
pwolanin commented@xjm - I agree. This code looks nice, but it's really not essential to solve the problem in a reasonable way.
Can we go back to the patch I marked RTBC at #27?
Comment #65
xjmAlso, note the change from #2506195: Remove SafeMarkup::set() from Xss::filter().
Comment #66
rainbowarrayYesCT asked me to take a look at this to see if I could help by bringing the patch from #24 based on pwolanin's comment in #64. I reviewed this issue and #2506195: Remove SafeMarkup::set() from Xss::filter(). As I understand things, we can no longer rely upon XSS:filter() as that no longer mark their results as safe.
The patch in #24 adds SafeMarkup::truncate(), but in #33, xjm suggests that SafeMarkup is not the right place to take care of that, so I'm not sure about bringing back #24 because of those concerns.
Any suggestions, xjm and/or alexpott, on how to proceed next?
Comment #67
chx commentedIf you want to truncate HTML then you need to split between tags and text and if you do that then you already did one half of the xss filter and since you want to display then you need to filter anyways so why not continue when you already half way there? That's the reasoning behind this patch. It works well and it's tested, I think we should use it.
Comment #68
pwolanin commentedLooking for similar uses - I only find one, which could be dangerous in that is decodes entities (though possibly the comment subject is escaped further)
\Drupal\comment\CommentForm
It's hard to say whether other uses of Unicode::truncate() should be removing HTML, but if we don't find any, perhaps we should just fix this case with an ugly one-liner, as opposed to adding an API method.
Comment #69
pwolanin commentedThis is a new, minimal patch.
Comment #70
joelpittet@pwolanin the approach @chx took is along the lines of your point: in #25
And also I'd like to get @effulgentsia's position on #61 as I think it resolves all worries brought up earlier.
Comment #71
pwolanin commented@joelpittet - effulgentsia was the one who originally suggested using strip_tags(). I think if we want to consider an API addition we should make a follow-up rather than dragging this issue out, since we didn't consider the breadth of use cases.
This issue is supposed to be about just removing SafeMarkup::set() and it's gotten way beyond that scope.
Comment #72
effulgentsia commentedI think #69 makes sense for this issue, because even in HEAD, we're not preserving any HTML tags in the truncated output. If at some point we have a need for a truncation function that preserves some HTML tags, then I think #61 can be posted to that issue to solve that use case.
Comment #73
genjohnson commentedI've read through the issue and summarized the proposed approaches and why they haven't been accepted.
Comment #74
genjohnson commentedI am reviewing and manually testing the patch from #69.
Comment #75
rainbowarraygenjohnson and I tried logging some messages in html_preprocess_html in Bartik to double-check the output.
We tried:
The output in the source on the log page was:
So it looks like this is doing what it should be doing. Actual HTML tags are being stripped out, and even if HTML entities when decoded could create a potentially problematic HTML tag, the l() function will prevent that HTML tag from showing up in source.
This looks like it works.
Comment #76
genjohnson commentedI read through the patch. The code makes sense, is in scope of the issue and seems to follow coding standards.
Comment #77
joelpittet@pwolanin @effulgentsia thanks for your replies, I'll open up a feature request for 8.1.x with @chx's patch.
The use-case would be search results body truncating with keeping the highlighting (strong or span) tags to highlight results. Maybe we are doing this another way, I'll try to confirm that still works for myself.
Comment #78
joelpittetYeah, FYI not problem on the search results, probably using a different approach.
Comment #79
yesct commented@joelpittet thanks for opening that separate issue.
It was #2544890: ::truncate() to allow certain tags to remain.
Comment #80
alexpottCan we add a test to prove that all markup is stripped.
Comment #81
xjm+1 for the current solution, and also for tests. ;)
Comment #82
stefan.r commentedI'll see about adding a test
Comment #83
stefan.r commentedComment #85
stefan.r commentedComment #86
rainbowarrayThe test message should probably be longer to ensure that it is a message that would be truncated.
Comment #87
stefan.r commentedWell it's currently supposed to be safe regardless of the length, are we testing anything specific with the truncation? According to #75 it would be OK to truncate anywhere for purposes of the test, so I'll just rewrite the test to do a lorem ipsum sentence instead of hello world then.
Just so everyone is clear, the current string:
<script>alert('foo');</script><strong>hello</strong> world"...is being converted into:
<script>alert('foo');</script>hello world...before being truncated (although the truncation does not apply to this string as it is too short), and is sanitized on output to:
<script>alert('foo');</script>hello worldComment #88
stefan.r commentedComment #89
rainbowarrayRevised test looks good!
Comment #90
alexpottCommitted 6cd78a1 and pushed to 8.0.x. Thanks!
Comment #91
alexpott