Closed (fixed)
Project:
Drupal core
Version:
8.0.x-dev
Component:
theme system
Priority:
Major
Category:
Task
Assigned:
Unassigned
Issue tags:
Reporter:
Created:
7 Jun 2015 at 13:26 UTC
Updated:
27 Apr 2016 at 03:35 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #1
peezy commentedComment #2
peezy commentedHave at it, testbot.
Comment #3
star-szrComment #4
star-szrGoing to review this.
Comment #5
joelpittetThe only other way we could do this is SafeMarkup::format(). Is there any reason why this was not done or considered?
It doesn't invoke twig so it should be considered as an option, IMO. But there may be a reason why it wouldn't work.
Comment #6
star-szrGreat point, so maybe this is relevant to #2501975: Determine how to update code that currently joins strings in SafeMarkup::set() actually.
Comment #7
joelpittetI think we can do it without that join because the items are finite. There are only ~4 pieces that need to be joined together. As long as we don't need to have arbitrary pieces that should be fine to join the pieces together.
Comment #8
star-szrRight but there were some arguments against doing
'@one@two@three@four'in another issue.Comment #9
joelpittetOh yeah, lol. Well then yes, let's postpone on that.
#2501975: Determine how to update code that currently joins strings in SafeMarkup::set()
Comment #10
alexpottActually I'm not sure that we actually need this SafeMarkup::set() here at all because whatever this returns is passed to the Twig engine which knows what to do with
Twig_Markup.Comment #11
alexpottThis change removes 4,300+ calls to SafeMarkup::set() on the user permissions page with 11 roles.
Diff%
Diff
(microsec)
Diff%
Diff
(microsec)
Diff%
MemUse
Diff
(bytes)
Diff%
MemUse
Diff
(bytes)
Diff%
PeakMemUse
Diff
(bytes)
Diff%
PeakMemUse
Diff
(bytes)
Diff%
Comment #12
alexpottWe can do even better... checking Twig_Markup in SafeMarkup::isSafe() saves the additional calls to SafeMarkup::checkPlain and htmlspecialchars.
Comment #13
alexpottComment #16
alexpottFixing some of the test fails.
Comment #17
alexpottFixed the views tests too.
Comment #19
alexpottThis should use the SafeString approach from #2506581: Remove SafeMarkup::set() from Renderer::doRender. Let's postpone this issue on that one.
Comment #20
dawehnerThis can be unpostponed now.
Comment #21
alexpottUsing SafeString now and removing other unnecessary calls to SafeMarkup::checkPlain in twig_render_template().
Comment #23
alexpotthmm... We need to preserve safeness across views rendering.
Comment #25
scotwith1tTaking a look at this at Drupal GovCon. Going to spend an hour or so looking at it...
Comment #26
scotwith1tHad trouble running tests locally. Uploading this to see what the testbot does with it. Not sure yet how to make this test pass, but changing the test may not be the best way to handle and may not even address the actual issue. Will look at it again tomorrow, but wanted to at least throw this at the testbot.
Comment #27
scotwith1tOops, forgot the files. :)
Comment #29
onelittlebecca commentedMe and Brandon were working on this at the govcon sprint. We believe the function that isn't being called is on line 1198 in file core/modules/views/src/Plugin/views/field/FieldPluginBase.php... going to run debugger tomorrow.
Comment #30
cilefen commented@pwolanin helped me narrow this down - please make sure he gets credited.
Comment #31
pwolanin commentedThe wrapping in SafeString and casting back is a little ugly for now, but avoiding adding a lot of strings to the safe string registry is important.
Comment #32
cilefen commentedNeeds work because we are merging the work on #2501745: Remove or document SafeMarkup::set in theme() in.
Comment #33
cilefen commentedComment #34
cmanalansan commentedMerged the work from #2501745: Remove or document SafeMarkup::set in theme()
Comment #35
pwolanin commentedWill the method ever return a string now?
Comment #36
cmanalansan commentedYes, there are times that it will still return a string. Either a blank string, or whatever is returned by $render_function().
Comment #38
cmanalansan commentedThe test needs to be adjusted to account for the fact that render() likely returns an object now. Working on this now.
Comment #39
cmanalansan commentedUpdated ThemeTest
Comment #41
davidhernandezis_string isn't quite the right check for the test. If something happens and a string gets returned, this condition is met. What we want is to check that it's an empty string.
The test fails were caused by a missing use statement.
Comment #42
pwolanin commentedThe else is nestled up here.
Also, I think it would be better to leave the is_sting() check and then use
assertIdentical($output, '')orassertTrue($output === '')Comment #43
yesct commentedlooking at the remaining tasks,
I think there is test coverage, cause we had to change it.
and 3. is not relevant cause it was refactored.
marking those two things done.
Comment #44
davidhernandezok
Comment #45
yesct commented(todo) we can remove the now unused use for SafeMarkup.
(nit todo) this could be wrapped closer to 80 chars.
(ok) I was wondering about casting this, seems like render() returns either a string or a SafeString object. So that must be why we are casting, so we are explicitly saying it is a string.
(nit todo) we are changing this line, so we can line wrap to 80 chars.
(ok) since this comes from editing a view, can be considered administrative text. so we are trusting it. just making the safe string object and not storing it helps not pollute the list of strings set as safe.
(question) in both of these cases, I dont understand the comment.
(todo) I think the return type needs to change on FieldHandlerInterface::renderText() to be string or the object (specify the interface for safe strings.
(question) are we always assuming that anything that has been rendered is safe? and so assuming that the string result of return $value . $suffix is safe also... then the comment about preserving if it is not safe is even more confusing. and maybe we should just make a safestring object in both cases?
(ok) looks similar to approach from #2506035: Do not add placeholders from SafeMarkup::format() to the safe list.
Comment #46
alimac commentedAddressing nitpicks 1, 2, and 4 from #45.
Comment #47
alimac commentedThis patch addresses the above.
Comment #48
alimac commentedComment #49
pwolanin commentedIn this case:
We don't really need to use a SafeString object at all, since the string is already on the safe list. I guess it doesn't hurt for consistency.
In the other cases that look like this, we are avoiding putting the concatenated string into the safe list since I guess we are close to the point of going to Twig. However, we need to make sure the $suffix is safe also - is that user input?
Comment #50
alimac commented@pwolanin, the $suffix doesn't seem to be user input, but then there is bit:
CoreUrl::fromUserInput?
Comment #51
alimac commentedPreserve whether or not the string is safe. If the string is not already marked safe, it will be sanitized by Twig.
^^ This should clarify points 6. and 8. in comment #45.
Comment #52
pwolanin commented@alimac in the link generator we have
so the link from \Druapl::l() should be safe.
Comment #53
alimac commentedLooking back at this issue... I see comments #21 and #23 from alexpott. A number of tests failed before SafeString code was added. So it is not just for consistency.
Comment #54
alimac commentedI took a stab at improving the comments, to address point 6. from #45.
Comment #55
yesct commentedthat makes much more sense. and addresses all my earlier concerns. as far as I can tell, this is ready.
Comment #56
alexpottWe need to preserve safeness here too.
Comment #57
alexpottThe patch in #56 wasn't right.
Comment #58
yesct commentedI think the change in #57 is #2505679: Remove SafeMarkup::set in FieldPluginBase::advancedRender() and so we can close 2505679 as duplicate of this issue.
The interdiff looks fine, just casting back to string, since now, we are returning a SafeString object.
Comment #60
RavindraSingh commentedDoesn't require to be added defined by string, as we are getting $this->last_render as a string.
Comment #62
alexpott@RavindraSingh at that point it might be a SafeString object. The problem this patch is coming up against is that we have have to ensure that the a SafeString is never cast to a string in views rendering and lose it's safeness.
Comment #63
alexpottViews does really funky rendering and uses the bubbled data to assemble the row in StylePluginBase::renderFields() which points out that we're doing the SafeString creation too late in Renderer::render(). (That was fun to work out).
Interdiff is back to #57 since, as explained in #62, #60 wasn't quite right.
Comment #65
alexpottFingers-crossed this will fix the rest of the tests.
Comment #66
alimac commentedI want to update the issue summary to reflect the most recent patch changes.
@alexpott, we are wondering why the new serializer was added?
Comment #67
alimac commentedComment #68
alimac commentedI found the failing test for which the new serializer was added:
Comment #69
alimac commentedComment #70
alexpottWe need the new serializer so exporting views using the serializer works. All this does is change SafeStringInterface implementing objects to strings... which is exactly what they would have been before this patch.
Comment #71
yesct commentedI'll review this.
Comment #72
alexpottIn this test (unlike others) I've kept the
assertIdenticalbecauseassertEqualsdoes not return the expected results when comparing things like 0 and '' so the actual text is very significant.Comment #73
pwolanin commentedA couple minor things to fix that I see.
This should probably be SafeStringInterface instead of the concrete class?
we should probably change the function @return doxygen?
Comment #74
yesct commentedLooked through the patch, and I agree aside from the things in #73 it is good, each change is needed for this issue (staying in scope).
Comment #75
alexpottThanks @YesCT and @pwolanin for the reviews. Here's are patch addressing #73
Comment #76
yesct commentedI think this should be string|\Drupal\Component\Utility\SafeStringInterface
Comment #77
yesct commentedlooking at the approach in the patch, updating the title.
Comment #78
alexpott@YesCT re #76 yep, in the hopefully extremely rare instance that we call twig_render_template to render an empty string :)
Comment #79
dawehnerWell, we should test against the thing which core returns, so are there examples which don't return a safe string?
So I guess this is done because if $object is always true basically? I'm wondering whether we could manipulate the logic inside twig to take into account SafeStringinterface. It seems quite a burden for themers to distinct that?
General question: Given now that safe string is used by a module and not only the internal systems, I'm curious whether the @internal flag on SafeString can still be justified as it is at the moment
Comment #80
pwolanin commentedPHP seems to be sadly lacking anything like a magic method to evaluate emptiness of object.
However, I think we'd have the same problem if children was a render array? Well, I guess an empty array would be empty here, but a render array with empty string markup would not be.
Comment #81
yesct commentedthe twig length change was in #17 to fix the Drupal\views\Tests\Plugin\RowRenderCacheTest fail in #16
Comment #82
alexpottRe #79
is not emptylikecore/vendor/twig/twig/test/Twig/Tests/Fixtures/tests/empty.test. The thing here is that we're not evaluating emptiness with{% if children %}Comment #84
alexpott@xjm has asked why views using SafeString in this patch. It's because all the code is fired by the rendering done in
StylePluginBase::renderFields()- where it calls$renderer->renderPlain($data);Comment #85
xjmI'm concerned about using
SafeStringInterfacein Views and about including #2505679: Remove SafeMarkup::set in FieldPluginBase::advancedRender() in the scope of this. @alexpott said that he found this to be necessary after 5h of debugging; I asked him to explain that a little more here on the issue.Comment #86
alexpottI can't reproduce this locally.
Comment #88
alexpottI'm pretty sure that we can limit all the SafeString futzing in Views to Drupal\views\Plugin\views\field\FieldPluginBase::renderText()... which helpfully complies with the current @internal directive on the SafeStringInterface since this method is a views-last-minute-whilst-rendering-stuff-change-everything-about method.
Patch also improves documentation when casting to string to explain why.
Comment #89
alexpottI guess we have outstanding questions about FieldPluginBase::renderTrimText() but atm this method is set up to completely lose safeness and that is not changed by this patch. It only has one usage in core which is FieldPluginBase::renderText() - which preserves safeness. I think we should protect this method and be done.
I've fixed the documentation of FieldPluginBase::theme() since in reviewing the interface I realised it's been wrong since rendering returns safestrings. It either returns an empty string or a safe string. I've also fixed FieldPluginBase::render() since this will also often return SafeStrings.
Comment #90
alexpottIt is possible to go further in
FieldPluginBase::renderText()and remove some of the SafeString::create()'s by providing methods for things that are obviously safe. For example, SafeString::trim() or SafeString::stripTags(). However doing this would break the immutability of the SafeString object so I think this is a bad idea. I can't think of any other option (at the moment) for how to not have to deal with SafeString creation in FieldPluginBase::renderText().Comment #91
xjmComment #92
alexpottEmpty strings should never be SafeStrings so actually I think the changes to the twig templates
{% if children is not empty %}are not necessary.Comment #93
dawehnerI'm curious whether at least one instance of the docs could explain, under which circumstances you get hack an object and when just a simple string.
Comment #94
alexpott#93 see SafeMarkup::create() ... it will a string when it is an empty string or an unsafe string. Whichever way it is the calling code should in general just treat it like a string. I guess we could add @see's everywhere to the sanitisation or render docs. But is it worth it?
Comment #95
alexpottre #93/#94 OTOH what's the harm. Added docs.
Also in discussion with @dawehner and @damiankloip making FieldPluginBase::renderTrimText() protected to reduce the surface area of the interface and make all render functions it has behave the same as core render functions which is to return a SafeStringInterface object where correct and possible.
I'm not sure that the API changes listed in the issue summary actually warrant a CR.
Comment #96
dawehnerYeah I'm not sure whether its worth, most code should work the same anyway, unless you do stupid things ...
+1 for making that method protected. The static issue is what actually matters as its kinda a useful tool
Comment #97
alexpottRerolled to use the new ViewsRenderPipelineSafeString object. I think this is good to go now.
Comment #98
wim leersJust one nitty question:
In all other assertions in this patch, the rendered output is cast to a string so that the identical comparison can remain. Why not also here?
This didn't even need to be changed, nice :)
Comment #99
alexpott@Wim Leers re #98.1 this is because identical-ness is super important to
FieldUnitTest. See #72.Comment #100
wim leersAlrighty :)
Comment #101
joelpittetThis touches so many things! And what is a ViewsRenderPipelineSafeString?
Comment #102
wim leers#101: that was added earlier today in #2544684: Expand @internal documentation on SafeString and SafeStringInterface and introduce ViewsRenderPipelineSafeString.
Comment #103
alexpottAdding some answers to why this patch touches so much code to the issue summary.
Comment #104
alexpottThis is now blocking #2547741: Introduce FilteredString and get rid of all SafeMarkup::set() calls in the Filter module and #2549791: Remove SafeMarkup::xssFilter() and provide ability to define #markup escaping strategy and what tags to filter
Comment #105
alexpottComment #106
alexpottComment #107
alexpott@xjm asked about test coverage of the changes in doRender and pointed out we're missing explicit test coverage. We only have implicit in Views testing. Patch attached fixes that. The test only patch is the interdiff :)
Comment #109
wim leersEven better :) RTBC++
Comment #110
catchJust a couple of nits which I fixed on commit, otherwise looks good.
Fixed on commit - XSS / example.
Comment #113
alexpottHere's a fix that allows us to run all the phpunit tests from the command line again. Also included @catch's fix on commit.
To test do
"cd core; phpunit --testsuite unit"from the command line.And this is reason #142 why SafeMarkup is very very tricky to work with.
Comment #114
alexpottSo testbot can not pick up failures like #113 because it runs each phpunit test individually in a separate process. The long and short of it is - static properties are bad and, yes, we knew that but it didn't stop us from using them anyway. /me dreams of drupal_static() from drupal since at least that allowed us to reset the lot.
Comment #115
wim leersNew testbot came back green. That means old testbot should come back green too. 50 minutes in, it's still waiting to begin testing. So expect that to come back green in ~25 minutes at best.
Comment #116
catchCommitted/pushed to 8.0.x, thanks!
Comment #119
extexan commentedThis issue was closed 9 months ago, and the patch was supposedly committed to core, but I'm on v8.0.5 and am still having this problem. The HTML markup I put in the "rewrite" text area is being shown.
I checked this patch (and a couple others that seemed to be related) and they do, in fact, appear to have been committed to core. However, the issue still remains. Any suggestions?
Comment #120
cilefen commented@ExTexan This issue was a refactoring task. The issue you are having sounds specific to Views and I think there have been a few double-escaping issues. Open a new issue in the Views component if none exists.
Comment #121
extexan commented@cilefen, Yes, will do. Thanks.
I just found the trigger is aggregation on or off. When off, HTML shows fine - when on, it is escaped.