Allow Twig in Views token replacement

Sounds like a pretty major bug.

I'll be working on this at the Amsterdam sprints.

@mikeker
Let's say at least hi :)

With all the changes around SafeMarkup::set(), I'd appreciate someone telling me if this is a valid way to fix this...

Note: unit tests are (in my opinion) outside of the scope of this issue because of #2346615: Views doesn't have multi-valued field unit tests.

Just talked with @dawehner and we agreed that this would be better fixed by rewriting Views' rewrite option to use inline Twig templates instead of a textarea powered by str_replace().

Patch coming soon...

Here's the quick plan after feedback from @joelpittet:

• Views UI remains basically the same: checkbox to override result with custom text, replacement values from this result row are available in square brackets (eg: [field_name])
• Add note in the UI that Twig formatting is available
• On render:
  • str_replace() the square bracket with double-curly brackets
  • Pass what remains to Twig as an inline template using the Views replacement patterns as a context

Ideally any D7 view that currently uses the rewrite option would work without change. But it adds the ability for people to use all the fun features in Twig, along with Twig's autoescaping and safe-by-default approach, to rewrite to their heart's content.

Going forward we could give the option to use a file-based template instead of an inline one.

Attached patch does what's outlined in #8.

Just noticed the tag for Amsterdam sprint issues...

Added tests so both D7 style ([field_name]) and Twig style ({{ field_name }}) rewrites are covered.

I tried last patch from #12, and it seems that with this patch the issue is fixed and it's working as described in #8.

To test patch I rewrote the "Body" field as <h2>[body]</h2>, which without patch resulted as:

<td class="views-field views-field-body" headers="view-body-table-column">
  &lt;h2&gt;&lt;p&gt;Body test&lt;/p&gt;&lt;/h2&gt;
</td>

and rendered as plain text in browser:

<h2><p>Body test</p></h2> 

With patch result is as expected:

<td class="views-field views-field-body" headers="view-body-table-column">
  <h2><p>Body test</p></h2>
</td>

and rendered in browser as expected:

Body test

Also tried with twig template syntax

<p>{% for i in 0..10 %}
    * {{ i }}
{% endfor %}</p>
{{ body }}

which resulted as expected:

<td class="views-field views-field-body" headers="view-body-table-column">
  <p> * 0 * 1 * 2 * 3 * 4 * 5 * 6 * 7 * 8 * 9 * 10 </p>
  <p>Body test</p>
</td>

and rendered in browser as expected:

 * 0 * 1 * 2 * 3 * 4 * 5 * 6 * 7 * 8 * 9 * 10
Body test

I really like the idea, but yeah let's skip the BC layer.

+++ b/core/modules/views/src/Plugin/views/field/FieldPluginBase.php
@@ -1306,14 +1306,34 @@ public function renderText($alter) {
+    // For backward compatibility, we convert square brackets to double-curly
+    // brackets so that "[field_name]" becomes "{{ field_name }}".
+    $replacements = [];
+    $context = [];
...
+      $token_name = substr($square_token, 1, -1);
+
+      // Twig cannot have '-' characters in variable names, but D7-style tokens
+      // can. Convert them to '_' for backward compatibility.
+      $token_name = str_replace('-', '_', $token_name);
+
+      $replacements[$square_token] = "{{ $token_name }}";
+      $context[$token_name] = $tokens[$square_token];
+    }
+    $template = strtr($template, $replacements);

Please let's us not take care about BC at this point.

@Wuk, Thank you for testing the patch and sharing your results.

@dawehner, I agree, backward compatibility makes the code a bit more complicated, but not overly so and I can't think of any way it would impact performance. And it simplifies the upgrade/migrate path. But I'm happy to pull it from the patch if you don't agree it's worth it.

If we remove backward compatibility, we would have to change Views' token system to replace [token-value] with {{token_value}}. Are there any other places in Views that we use token replacement? (I can't think of any off the top of my head...) If there are, perhaps we should convert these to Twig-style rewrites as well? It would be good to get all of them at once!

If we remove backward compatibility, we would have to change Views' token system to replace [token-value] with {{token_value}}. Are there any other places in Views that we use token replacement? (I can't think of any off the top of my head...) If there are, perhaps we should convert these to Twig-style rewrites as well? It would be good to get all of them at once!

Mh you are right ... there are though places like in Entity area handlers where you can use the tokens coming from arguments, mhhhhhhh

So let's remove the BC and use different kind of tokens for fields than for the other stuff.

Looks like a duplicate of #2337747: HTML entities/tags in the item title are double encoded for feeds using fields, has the same fix from what I can see.

Looks like a duplicate of #2337747: HTML entities in the item title are double encoded for feeds using fields, has the same fix from what I can see.

Huch? This is not the same fix ... given that this issue introduces the suage of twig inside views.

Added issue summary.

Added Beta Eval to the issue summary.

Attached patch aims to add Twig to all tokenized fields in Views. Work-in-progress, hoping to get some feedback from the testbot and others.

Moved the token replacement code into PluginBase and fixed a bunch of tests (updating [token] to {{token}}). Replaced most (all, hopefully) of the hidden strtr() token replacements hidden throughout the code.

Helps to attach the patch...

Fixes the last failing test.

Re. #18/#19: Sorry yes, the first patch was the same, now we're going for a different solution. Anyway, my conclusion is still correct I think, if we replace that initial fix with the twig replacements, then it might also fix the other issue which needed the same fix?

@Berdir: now that the testbot is green, I'll see if it fixes #2337747: HTML entities/tags in the item title are double encoded for feeds using fields and a few other rewrite issues I saw in the issue queue.

I did same as in #13 using twig syntax and it works as expected.

@Berdir: No dice... The patch does not fix the issue raised in #2337747: HTML entities/tags in the item title are double encoded for feeds using fields, unfortunately.

There is also #2280961: (Views)FieldPluginBase::advancedRender() calls SafeMarkup::set() on a string that it doesn't know to be safe, not sure how this is related :)

There is also #2280961: Views markup and SafeMarkup, not sure how this is related :)

That other issue is the general meta issue to get rid of all of them.

This issue fixes a particular subset of existing problems.

+++ b/core/modules/views/src/Plugin/views/PluginBase.php
@@ -293,6 +293,57 @@ public function globalTokenReplace($string = '', array $options = array()) {
+  public function viewsTokenReplace($text, $tokens) {

Just curious, is there a need to have this method to be public?

@dawehner: Probably not... Protected would be just fine. Will adjust the patch when I get back from holidays.

Changes public to protected as @dawehner suggestd in #34. Also cleaned up some doc typos.

HA! I did the interdiff backwards... :)

Use this one.

sigh... Rerolled against the latest HEAD.

Alright, well let's get this in, better sooner than later.

Does this need a change record?

In case we do, I started a draft change record.

I fear :(

@dawehner: details?

OH I guess this was a comment conflict :)

Committed 552c86c and pushed to 8.0.x. Thanks!

Thanks for adding the beta evaluation to the issue summary.

Just want to say thanks to everyone who worked on this or helped out in any way, especially @mikeker and @dawehner! This is really exciting, to be able to use Twig filters and other Twig basics from Views UI seems very cool to me.

@Cottser: I appreciate the kind words.

I'm really hoping this allows site builders all sorts of flexibility. The first such idea is showing up in #2055597: Allow multi-value field rendering to pass through Views rewrite only once, which started out as a request to allow different HTML around single values in multi-valued fields. There is a security concern in that issue that I would appreciate feedback on. In short, Xss::filterAdmin encodes '<' and '>' which causes Twig to barf if you have greater/less-than conditionals in your rewrite.

Need some feedback over in #2396607: Allow Views token matching for validation (and remove dead code):

+++ b/core/modules/views/src/Plugin/views/PluginBase.php
@@ -320,6 +320,57 @@ public function globalTokenReplace($string = '', array $options = array()) {
+      if (strpos($token, '{{') !== FALSE) {
+        // Twig wants a token replacement array stripped of curly-brackets.
+        $token = trim(str_replace(array('{', '}'), '', $token));

Twig supports lots and lots of things with the {{ }} operators, like:
{{ "foo #{1 + 2} baz" }}

However, the code above would silently break that. Should Views only support plain tokens like {{ foo }}? Or maybe there is a usecase to support {{ foo.bar }} or richer expressions like:
{{ user.username|e('css') }}

#2396607: Allow Views token matching for validation (and remove dead code) is adding a bit of API for Views token handling, so it would make sense to address this over there.

Filed a minor UI string fix: #2414685: Improve references to Twig tokens in Views UI

1. +++ b/core/modules/views/src/Plugin/views/PluginBase.php
   @@ -320,6 +320,57 @@ public function globalTokenReplace($string = '', array $options = array()) {
   +    foreach ($tokens as $token => $replacement) {
   +      if (strpos($token, '{{') !== FALSE) {
   +        // Twig wants a token replacement array stripped of curly-brackets.
   +        $token = trim(str_replace(array('{', '}'), '', $token));
   +        $twig_tokens[$token] = $replacement;
   +      }
   +      else {
   +        $other_tokens[$token] = $replacement;
   +      }
   

   Post-commit review:

   This feels wrong, because while it works for simple cases, twig has many ways to get a variable and this would not allow those.

   Proposal:

   We know that a token is either %1 or @1, so replace those directly.

   Push as context all tokens except those tokens directly.

   Nice to have:

   Or maybe even make them available as a tokens variable.

   So I could write e.g.:

   {{ args[1] }} for %1

   and

   {{ tokens[1] }} for @1 or such?

2. +++ b/core/modules/views/src/Plugin/views/PluginBase.php
   @@ -320,6 +320,57 @@ public function globalTokenReplace($string = '', array $options = array()) {
   +    // Non-Twig tokens are a straight string replacement, Twig tokens get run
   +    // through an inline template for rendering and replacement.
   +    $text = strtr($text, $other_tokens);
   

   Overall the straight string replacement while good for keeping BC in a way is a security nightmare.

   Because if I do:

   %1{%2

   e.g. I might suddenly - given enough input possibilities - be able to dynamically change the twig template.

   And that is "Twig-Injection-Exploit"?

   I know it might not be possible, but if framework manager would allow breaking BC, the best would be to just use:

   {{ tokens['%1'] }}
   {{ tokens['@1'] }}

   instead or find some shorter variable names like:

   {{ at[1] }} -- @1
   {{ p[1] }} -- %1

   ===

   This also should at least use the new SafeReplace function from the SafeMarkup for #markup patch - once it is available.

3. +++ b/core/modules/views/src/Plugin/views/field/Field.php
   @@ -890,7 +890,7 @@ function render_item($count, $item) {
   -      $tokens['[' . $this->options['id'] . '-' . $id . ']'] = $this->t('Raw @column', array('@column' => $id));
   +      $tokens['{{ ' . $this->options['id'] . '-' . $id . ' }}'] = $this->t('Raw @column', array('@column' => $id));
   
   @@ -908,11 +908,11 @@ protected function addSelfTokens(&$tokens, $item) {
   -        $tokens['[' . $this->options['id'] . '-' . $id . ']'] = Xss::filterAdmin($raw[$id]);
   +        $tokens['{{ ' . $this->options['id'] . '-' . $id . ' }}'] = Xss::filterAdmin($raw[$id]);
   ...
   -        $tokens['[' . $this->options['id'] . '-' . $id . ']'] = '';
   +        $tokens['{{ ' . $this->options['id'] . '-' . $id . ' }}'] = '';
   

   I think it would be enough to just have e.g.:

   $tokens[$this->options['id']] = Xss::filterAdmin($raw[$id]);

   and not deal with either '[]' or '{{' to support my comment above.

Per Alex' request posted as follow-up here: #2492839: Views replacement token bc layer allows for Twig template injection via arguments

This issue is in imminent danger of being rolled-back prior for RC1, because #2492839: Follow-up: Convert Views' %n and !n replacement tokens to Twig syntax has not been addressed (according to #15 in that issue).