The issue summary of #2567257: hook_tokens() $sanitize option incompatible with Html sanitisation requirements says that Token::generate() should:

Always return SafeString as-is, or escape raw strings

But the committed patch doesn't do that. Here's a follow-up that does.

CommentFileSizeAuthor
token-generate.patch3.57 KBeffulgentsia

Comments

effulgentsia created an issue. See original summary.

catch’s picture

Priority: Normal » Major

Ouch, nicely caught.

effulgentsia’s picture

+++ b/core/lib/Drupal/Core/Utility/Token.php
@@ -268,6 +264,12 @@ public function scan($text) {
+   * Typically, application code does not invoke this function directly, but
+   * rather invokes the replace() function, and this gets called via that.
+   * However, application code that wants to retrieve token values in isolation
+   * rather than merged into a string with token placeholders may use this
+   * function to do so.

I was curious if there were any real use cases of Token::generate() being called by anything other than hook_tokens() implementations themselves, and here's a couple that I found:

  1. CTools context plugins, such as in ctools_context_node_convert(). But I don't see a D8 equivalent of this in the 8.x branch of CTools, perhaps because CTools contexts have been refactored for 8.x around TypedData API?
  2. Within Ubercart, there's a usage in a preprocess function: template_preprocess_uc_packing_slip().

Not sure if we want the docs here expanded to explain such use cases and/or expanded to also mention that hook_tokens() implementations call generate() for their nested tokens.

alexpott’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Let's get test coverage for this.

dawehner’s picture

Interesting, well I always thought Token::generate() was just public because its also used in some hook_tokens() implementations.

catch’s picture

If hook_tokens() call it, wouldn't that result in double escaping if we move the sanitization to Token::generate() - Html::escape() twice.

dawehner’s picture

Yeah I think so.

effulgentsia’s picture

No, because of the SafeString::create() in:

+++ b/core/lib/Drupal/Core/Utility/Token.php
@@ -318,6 +324,11 @@ public function generate($type, array $tokens, array $data, array $options, Bubb
+      $replacements[$token] = SafeMarkup::isSafe($value) ? $value : SafeString::create(Html::escape($value));

The last submitted patch, token-generate.patch, failed testing.

dawehner’s picture

Ah perfect, well, then we could easily move it to Token::generate(), sorry for the noise.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

adamps’s picture

It's not always the case that a token will be used to form markup. For example, they could be used for email headers, or to make a URL (see RRSSB module). Or the value might be some intermediate stage that after further calculations would later be converted to markup. In #2580723: Fix token system confusion, with new function Token::replacePlain() we helped clarify these cases by adding new methods replacePlain() and replaceMarkup().

Therefore if we consider that generate() is part of the public interface, then I propose that it should return the token values as-is. There is no ambiguity, because the markup once implement MarkupInterface. There is no security risk, because Drupal has mechanisms to auto-escape a plain text value (e.g. in a twig template).

This fix as it stands would be a BC-break. If any code is calling generate() then using the result in a plain-text context, then HTML escaping could corrupt the value. For example if a module added a handy token [user:mail_display], which returns something like Adam<adam@example.com> then it certainly doesn't want to be escaped.

I propose this issue could be marked as "Closed (works as designed)"

darvanen’s picture

I tend to agree. I'd like a third opinion though.

larowlan’s picture

I'll ping @effulgentsia and ask if he agrees to close this in favour of #2580723: Fix token system confusion, with new function Token::replacePlain()

andypost’s picture

Tests should cover translatability of markup, ++ to new methods and deprecate old ones for D11

In D10 under PHP 8.0 it could use https://wiki.php.net/rfc/stringable which has polyfill in Synfony

mxh’s picture

Or the value might be some intermediate stage that after further calculations would later be converted to markup.

This is exactly how I'm using the Token::generate() method in Mustache Templates. Moving the sanitization part to Token::generate() would get me into trouble, possibly making the original core service unusable and I'd need to either replace that service by my own or decorate it (once more). Thus +1 for #19..

ECA heavily relies on the Token service as well, there it's being used for many different purposes, including for variable assignments. Thus changing the behavior of ::generate() might likely also make the service harder to use in that regard.

mxh’s picture

Version: 8.9.x-dev » 9.3.x-dev
darvanen’s picture

@larowlan I'm guessing there was no response from @effulgentsia?

I propose we close this ticket based on the recent feedback with the caveat that they can reopen it if there's a need.

larowlan’s picture

He's been busy with CKEditor 5, I'll follow up

effulgentsia’s picture

Status: Needs work » Closed (won't fix)

Regardless of the original intent of this issue, I agree that at this point, we should not break BC of Token::generate(), so closing this.

I haven't yet reviewed #2580723: Fix token system confusion, with new function Token::replacePlain() (and no need to delay that on me if others are happy with it) to know if it addresses the underlying problems that prompted this issue, but even if it doesn't, it doesn't have to be that issue's responsibility to do that.