Support multiple instances of the same token type in token replacement

When #1968214: 'unify' hook_token_info() gets in, we can require data that is passed on to token_replace() to have a "type" parameter which refers to any of the types exposed in hook_token_info(). This way, data names and types are separated, and with a few modifications we can easily pass on an arbitrary number of data objects of the same type to token_replace().

My apologies for posting that. It was a quick idea I had right before I had to leave and I posted it without checking your patches, which is what I did just now.

I understand the approach you're taking here. I haven't tested the patched but I believe it should work well. I have two questions:
1) What if the type of the data cannot easily be obtained, such as when the data is not an object?
2) Does this mean that we can remove the data type from needs-data in a follow-up, as we are using a different method of determining datas' types now?

I wonder if we should not use $entity->entityType rather than instanceof to determine an entity's type.

Is this what Drupal 6 did with token_replace_multiple()? It's not possible in D7?

Interesting re-roll ;)

Added a new patch with the approach described in #2367105: Support token alias (multiple instances of the same token type) for multiple token instances.

In short this patch adds token alias support. An example:

$token_service = \Drupal::service('token');

// Load a nodes then replace tokens in the text.
$text = 'The first node is [node{first}:title] and the other node is [node{second}:title]';
$node1 = node_load(1);
$node2 = node_load(2);

$data = array(
  'node{first}' => $node1,
  'node{second}' => $node2,
);

$output = $token_service->replace($text, $data);

Internally the code extracts the different alias and call Token::generate using a certain context that includes:

• The token alias stripped so modules implementing token_info don't need to know about token alias. In the previous example the first context includes the type 'node' (instead of node{first}).
• The $data filtered so only the data related to current alias is passed. In previous example only $node1 is passed under the 'node' key.

The patch is not complete because it doesn't handle the hook_tokens_alter(), but the appoach can be easily extended to cover this case. It can be optimized also. But first let's see if this is a good approach or there are issues that I'm not aware of.

This is a good idea, but after a while (a couple of years!), the last patch is not applying anymore, so marking as NW.

Rerolled!

Not including interfiff because file have been moved.

The patch is just porting the previouspatch code to the new Token.php file but adding the new BubbleableMetadata parameter when calling replace.

Again, this is just a proof of concept, there's much room for improvement.

BTW, now that token system is a service, I suppose a module could implement this idea and provide a modified token service, right? It'd have to extend the Token class.

It seems is not needed to modify the regexp, in fact TokenScanTest fails if yes.

With original regexps multiple token support works ok.

An example of how to use it:

$token_service = \Drupal::token();
$string_with_tokens = 'First node: [node{one}:title]........ another node: [node{two}:title]';

$node1 = node_load(1);
$node2 = node_load(2);

$string_with_tokens_replaced = $token_service->replace($string_with_tokens, array('node{one}' => $node1, 'node{two}' => $node2));
dpm($string_with_tokens_replaced);

The type{name} syntax seems very clunky. Why is it even necessary? Every entity contains its metadata, right? So Token would know $data['client'] is a User, and $data['node2'] is a Node. Why not parse all tokens and look for their types in $data?

Token supports more than entities but also things like dates and any other data structure.

The patch contains a ksm(), that's why everything explodes.

Ok, big fail with last patch.

Let's rework the regexps. Standard scan should not match the aliased tokens, and alias scan should not match empty type tokens (for example: [:empty type token])

Notice: Undefined index: user{my_alias} in Drupal\Core\Utility\Token->replace()

Hi, I get this error if add an explication of how to use this in a field description.

I attach a new patch with a solution that works for me.

Rerolled for 8.3.x-dev

The type{name} syntax seems very clunky.
-- @rudiedirkx

Token supports more than entities but also things like dates and any other data structure.
-- @berdir

Could we get away with just aliasing in the token string, and passing the data with arbitrary names?

$source_string = 'First node: [node:title]........ another node: [node{other_node}:title]';
$replaced_string = \Drupal::token()->replace($source_string, [
  'node' => node_load(1), 
  'other_node' =>node_load(2),
];
dpm($replaced_string);

That way, the code offering multiple same-type parameters can just "dump" them into the token service. Example scenario: Passing both the current and specific node revision straight from \Drupal::routeMatch()->getParameters()->all() on /node/{node}/revisions/{node_revision}/view (Route: entity.node.revision).
The user would still need to know that [node:*] and [node_revision:*] tokens are available, and what data type they are (so the correct token type is used). But code wouldn't need to change if additional data is already passed in.

I've also made some updates for code style.

I confirm #44 is working nicely and apply cleanly

#44 is a two-year old patch for Drupal 8 without tests, there really is no point in RTBC'ing that as there is no way that will get committed like that. This just generates work for maintainers :)

Berdir, if the functionality is ok for maintainers I'll be happy to work on this. Currently, I don't know if is interesting, mainly because it got little attention. I'll try to address any comments (like adding tests).

However, the first thing to do is confirm the approach is ok and optimize the code. Given that it changes (or extends) how tokens are shaped I think we need some discussion here.

Yeah, it's though to bring any token related changes in core.

FWIW, it's a bit of a chicken/egg situation in regards to test and documentation versus discussion. IMHO, having test coverage would work very well as examples and basis for such a discussion. If we end up changing the syntax then it will take a few minutes to update examples in a test string. And it will help to verify that the regular expressions are working correctly.

So if you do want to bring this forward, then I'd strongly suggest to work on tests.

Ok, thank you very much for your comments Berdir, I'll focus on tests.

Changed implementation to a simpler one (approach is the same but I don't why the previous implementation was so complex).

Tests added!

Fix coding standards.

A missing comma that triggered a coding standards error.

Fixing wrong function name in @cover clause.

@tunic There is a bug in #56... it can't work without the 2 following changes

    foreach ($text_token_aliases as $type => $text_token_alias) {
      $data_for_current_token = [];
      $unaliased_tokens = [];
      foreach ($text_token_alias as $tokens) {
        foreach ($tokens as $token => $aliased_token_name) {
          $unaliased_token_name = "[$type:$token]";
          $data_for_current_token[$unaliased_token_name] = $data[$aliased_token_name];
          $alias_tokens_mapping[$unaliased_token_name] = $aliased_token_name;
          $unaliased_tokens[$token] = $unaliased_token_name;
        }

        $replacements_unaliased = $this->generate($type, $unaliased_tokens, $data_for_current_token, $options, $bubbleable_metadata);

        foreach ($replacements_unaliased as $unaliased_token_name => $value) {
          $aliased_token_name = $alias_tokens_mapping[$unaliased_token_name];
          $replacements[$aliased_token_name] = $value;
        }
      }
    }

should be

    foreach ($text_token_aliases as $type => $text_token_alias) {
      $data_for_current_token = [];
      $unaliased_tokens = [];
      foreach ($text_token_alias as $alias => $tokens) { // ADDING $alias as the key
        foreach ($tokens as $token => $aliased_token_name) {
          $unaliased_token_name = "[$type:$token]";
          $data_for_current_token[$type] = $data[$alias]; // HERE IS THE MAIN CHANGE
          $alias_tokens_mapping[$unaliased_token_name] = $aliased_token_name;
          $unaliased_tokens[$token] = $unaliased_token_name;
        }

        $replacements_unaliased = $this->generate($type, $unaliased_tokens, $data_for_current_token, $options, $bubbleable_metadata);

        foreach ($replacements_unaliased as $unaliased_token_name => $value) {
          $aliased_token_name = $alias_tokens_mapping[$unaliased_token_name];
          $replacements[$aliased_token_name] = $value;
        }
      }
    }

#58, the original token is stored in the $text_token_aliases and it is part of the return value of the alias_scan function.

I guess you were confused by the alias_scan documentation because it was wrong. I've updated the patch fixing the documentation.

#59: No, when using the code without correction I got a warning at this line:
$data_for_current_token[$unaliased_token_name] = $data[$aliased_token_name];
And the aliased replacement didn't occurs.
After the 2 proposed corrections in #58, the code runs fine and the (aliased) replacements occurs.
Maybe you can use a debugger and you will see that the index doesn't exists in this part of the mentioned lined $data[$aliased_token_name];

#60, can you provide the test case you are using?

Something like this:

    $source_text="<p>Title1: [node:title]<p>Title2: [node{pc}:title]";
    $data=[
      'node' =>Drupal\node\Entity\Node::load(1),
      'pc'=>Drupal\node\Entity\Node::load(2)
    ];
    $options = [
      'clear' => FALSE,
      'langcode' => 'fr',
    ];
    $replaced_text = Drupal::token()->replace($source_text, $data, $options);
    dpm($replaced_text);

You can paste this code in the devel "Execute PHP code" page after having replaced the 2 IDs of the nodes by your existing ids.

Thanks, DuneBL, indeed it was wrong. In fact, everything was wrong, even the tests.

I've refactorized the code to fix it, including proper tests. Thi include a change on the format of alias_scan to allow a small simplifcation on the replace function.

DuneBL, keep in mind that when you use an aliased token you need to provide the type and the alias in the data array. So, instead
'pc'=>Drupal\node\Entity\Node::load(2) it should be 'node{pc}'=>Drupal\node\Entity\Node::load(2) .

Your test case would be:

    $source_text="<p>Title1: [node:title]<p>Title2: [node{pc}:title]";
    $data=[
      'node' =>Drupal\node\Entity\Node::load(1),
      'node{pc}'=>Drupal\node\Entity\Node::load(2)
    ];
    $options = [
      'clear' => FALSE,
      'langcode' => 'fr',
    ];
    $replaced_text = Drupal::token()->replace($source_text, $data, $options);
    dpm($replaced_text);

With the last patch, this test case runs ok!

Thank you!
Looks like it is working well.

This can't be RTBC when tests are failing or more specific, not running due to coding standards.

Sorry Berdir, I totally forgot about coding standards, shame on me.

Quick re-roll attempt as it no longer applies to 9.4

Re-rolled patch applied successfully and cleanly.

Tests didn't pass.

One more attempt

Open thread in the MR.

Also why in the MR did you add phpstan-baseline update? Don't see that in the patch before.

@smustgrave

Updated baseline due to this PHPStan error https://www.drupal.org/pift-ci-job/2558692

Running PHPStan on *all* files.
 ------ ---------------------------------------------------------------------- 
  Line   core/lib/Drupal/Core/Utility/Token.php                                
 ------ ---------------------------------------------------------------------- 
         Ignored error pattern #^Variable \$aliased_token_name might not be    
         defined\.$# in path                                                   
         /var/www/html/core/lib/Drupal/Core/Utility/Token.php was not matched  
         in reported errors.                                                   
  269    Variable $alias_tokens_mapping might not be defined.                  
 ------ ---------------------------------------------------------------------- 

That seems like a valid warning that may need to be addressed vs ignored.

Would have to compare to previous patch though and see what’s different in the MR.

Agreed, new problems should not be ignored but fixed.

Hi, for everyone coming to this issue after 10+ years, I have created a small Token extension module, were aliased tokens can be used in a programmatically way. It is a different approach, in syntax, but if you are desperate you might give it a try.

https://www.drupal.org/project/token_aliases

(You may have to use it from the git directly.)

Usage:

  $token_service = \Drupal::service('token_aliases');
  $string = 'Hello [user1:username], we have news from [user2:username] see [node:title] and [my_node:title] for more info.';
  // Fully loaded Drupal entities which are keys with the aliases used in the string.
  $data = [
    'user1' => $user1,
    'user2' => $user2,
    'node' => $node,
    'my_node' => $my_node,
  ];
  $result = $token_service->plainReplace($string, $data);

There's a bug with the latest MR. The variables should be reset a loop deeper.

I do not have time to make a patch or MR, but the main block should look like this:

    foreach ($text_token_aliases as $type => $text_token_alias) {
      foreach ($text_token_alias as $alias => $tokens) {
        // Reset the variables in this loop.
        $alias_tokens_mapping = [];
        $data_for_current_token = [];
        $unaliased_tokens = [];
        foreach ($tokens as $token => $unaliased_token_name) {
          $unaliased_tokens[$token] = "[{$unaliased_token_name}]";
          $alias_tokens_mapping["[{$unaliased_token_name}]"] = "{$type}{{$alias}}:$token";
        }
        // @todo Check if need to add else condition ?
        if (isset($data["{$type}{{$alias}}"])) {
          $data_for_current_token[$type] = $data["{$type}{{$alias}}"];
          $replacements_unaliased = $this->generate($type, $unaliased_tokens, $data_for_current_token, $options, $bubbleable_metadata);

          foreach ($replacements_unaliased as $unaliased_token_name => $value) {
            $aliased_token_name = $alias_tokens_mapping[$unaliased_token_name];
            $replacements["[{$aliased_token_name}]"] = $value;
          }
        }
      }
    }

Otherwise you get very weird issues. Easy way to test:

function test_token_reps() {
  $data = [
    'user{target}' => User::load(1),
    'user{source}' => User::load(0),
    'user{thirdy}' => User::load(6),
  ];

  $text = "
  Target UID: [user{target}:uid]
  Target Name: [user{target}:name]
  Source Name: [user{source}:name]
  Thirdy UID: [user{thirdy}:uid]
  Source Name: [user{source}:name]
  Target Mail: [user{target}:mail]
  Thirdy Mail: [user{thirdy}:mail]
  ";

  $text = Drupal::token()->replace($text, $data);

  echo $text;
}

The thing is that you can't use the same fields for all the aliases, gotta have some diffs. Then it goes haywire.

I have created a new MR (https://git.drupalcode.org/project/drupal/-/merge_requests/8706) with the following changes:

- Changed the test to return different tokens depending on the passed entity so we don't rely on the onConsecutiveCalls() approach. IMO the new approach is more consistent.

- Reseted the $alias_tokens_mapping array on the inner loop as suggested in #82

Adding a patch with the latest status of https://git.drupalcode.org/project/drupal/-/merge_requests/8706

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Hiding patches from bot.

MR should be updated to 11.x also as current development branch.

Thanks for the feedback, smustgrave. On it.

I have changed the base branch from the issue. I'm going to change the base dev branch from the MR as well

Created a new MR based on 11.x and fixed the bot reported problems: https://git.drupalcode.org/project/drupal/-/merge_requests/8707

The pipeline finally passed. This patch matches the latest state of the MR https://git.drupalcode.org/project/drupal/-/merge_requests/8707 with the pipeline in green.

Ran the test-only feature https://git.drupalcode.org/issue/drupal-1920688/-/jobs/2066476 which shows the coverage.

Think instead of updating the baseline though should fix the error around alias_tokens_mapping, maybe setting a default like

$data_for_current_token = [];
$unaliased_tokens = [];