Without reading the 522 long field tokens thread where this may already be covered, what are your thoughts about adding this option?

I've hit this requirement a lot recently, two examples are:

Webform custom email component
By using tokens and webform 7.x-4.x it is possible to add an email field on the node and embed this into the webform to send to that email address on submission. Currently, if you use the email field for this, you need to define this field as plain text in the token display mode. More of a pain than anything else having to define this custom display mode and configure this field not to use the default formatter.

Read More Control integration
This module allows you to pull in entity tokens for the link, but the setting allows HTML so users have full control of the "Read more<span class="element-invisible"> about @title</span>", so you can specify a field here, "[node:field_read_more]<span class="element-invisible"> about [node:title]</span>". Since the token uses the safe value, this encodes HTML entities, et al, even though the token_replace() has the 'sanitize' setting set to FALSE.

Both these would be fine using the raw values. In the second use-case, the raw value is required.

One possible solution is to define a new hook that defines a raw field value. If missing, you could drop back to the $item['value'] key if set. A possible hook for a file field could be (throw a coin if to decide if this one should use the fid or file path, file path seems more intuitive to me)

function file_field_token_raw($items, $context) {
  $values = array();
  foreach ($items as $delta => $item) {
    if (isset($item['file'])) {
      $values[$delta] = $item['file']->filename;

  return $values;
#3 field_tokens-sanitize-1713164-3.patch2.14 KBTessa Bakker
FAILED: [[SimpleTest]]: [MySQL] 63 pass(es), 180 fail(s), and 7 exception(s). View
Members fund testing for the Drupal project. Drupal Association Learn more


dww’s picture

Category: feature » task
Priority: Normal » Major

Yes, we need access to raw field values via tokens for modules like auto_nodetitle to work. See #1071596: HTML Special characters in node title for lots of confusion on the topic.

For text fields, it seems you can configure the 'tokens' view mode, and configure them to use 'plain text' and it basically works. But then that token is *always* unsanitized. It'd be nice if you could just conditionally express this when you use the token, instead of having to choose site-wide if a given field's tokens will be safe or raw...

I tried to follow most of #691078: Field tokens but it wasn't obvious to me why the raw tokens were removed. Seems like a separate issue here for this request is valuable.

Also, this *seems* like a regression from D6 token.module, so I'm at least going to call it a "task", not a "feature". Seems more like "bug" to me, but I don't want to piss off Dave Reid. ;)


Media Crumb’s picture

Did anything ever happen with this. I'm running into an issue with Meta tag module where I need to get access to the RAW token value. Pulling my hair out. Not sure why these values show up in views just fine.

Tessa Bakker’s picture

Title: Raw field tokens » field_tokens() isn't respecting sanitize option
Category: Task » Bug report
Status: Active » Needs review
2.14 KB
FAILED: [[SimpleTest]]: [MySQL] 63 pass(es), 180 fail(s), and 7 exception(s). View

The module auto_nodetitle uses the option sanitize = FALSE, but this isn't respected by field_tokens().
Therefore the module does some tinkering to fix this.

It would be great if this bug can be fixed in tokens, so other modules can relay on field_tokens() again.

A possible solution is attached as a patch, but it depends on Entity.

Status: Needs review » Needs work

The last submitted patch, 3: field_tokens-sanitize-1713164-3.patch, failed testing.

Dave Reid’s picture

Category: Bug report » Feature request

We cannot just add a late dependency on entity module. It would have to be completely optional please.

Given that Field API itself provides no options for unsanitized vs sanitized, reclassifying as a feature request.

Alan D.’s picture

So if it currently renders HTML version with sanitized content... maybe a post-processor to strip the tags and to un-sanitize?

i.e. something like this totally untested code:

$field_output['#post_render'][] = 'token_post_render_field_token';

function token_post_render_field_token($markup, $elements) {
  // Strip tags and de-sanitize if required.
  // using  empty($elements['#token_options']['sanitize']) as it is more obvious than !$elements['#token_options']['sanitize']
  if (isset($elements['#token_options']['sanitize']) && empty($elements['#token_options']['sanitize'])) {  
    $markup = strip_tags($markup);
    // This is PHP 5.1.0 + function. DRUPAL_MINIMUM_PHP = 5.2.4 in 7.35
    $markup = htmlspecialchars_decode($markup);
  return $markup;
Tessa Bakker’s picture

Nice idea, but you can't un-sanitize a sanitized string.
In the end I like to see every html tag or any other great xss code in my raw field result.

Would a sub-module be oké?

Let say the sub-module 'tokens_raw' has a dependency on Entity and the field_tokens() handles the raw function if the sub-module is active. In this way the 'documentation' part is fixed for beginners and the functionality works as expected.

Reading code in the morning isn't a good idea. :)
@Alan D. Will try your post-processor solution this week!

Alan D.’s picture

Let say the sub-module 'tokens_raw' has a dependency on Entity and the field_tokens() handles the raw function if the sub-module is active.

hehe, I have used this approach regularly... a custom token that does what I want