Static caching for token processing; iconv_substr() is super slow

The token replacement is in core, node_tokens() is in core too. And static caching is tricky, because based on what/how do you invalidate it exactly? the same node could suddenly have different values?

Maybe it's metatag module that should do the caching here? You're the one calling get_tags_from_node() twice for the same node? :)

Also, it looks like you're relying on the symfony mb_strlen() polyfill, which certainly isn't going to help with performance. You should enable the mbstring extension.

Feel free to raise the issue in core, but I think this is up to the caller, aka you.

Hi @DamienMcKenna,

I am curious the status of this issue? I see not much activity has occurred here, too.

Here's a simple patch that adds static caching for each unique value that is passed to Token::replace().

In my case, that's saving 12 Token::replace() calls, although not all of them actually have tokens.

Savings aren't huge, but if you have more identical tokens they could easily be bigger. It's fairly common to reuse the same tokens many times, e.g. for title, og title, twitter title.

Didn't check for $entity being NULL.

Here's another profiling.

Before:

After:

51ms.

Token generation leads the mass.gov performance report (and thats not a good place to be). I cant say if this patch is a good idea but I do know the need is large.

That's not exactly very useful feedback :)

Do you have any numbers on how much "leading" exactly is, how much this patch helps and also, have you seen my comments on e.g. #2935187: Token parsing massively slows down Drupal 8; document that mbstring is recommended and #3039650: Metatag discards cacheability metadata, results in a performance hit on how to speed up token generation? one of the most important things is to use property tokens, so :value and so on, that apparently saved 37% for @johnwebdev.

How much this patch benefits will depend directly on how many duplicated metatag tokens you're using.

I'm not following this issue closely (i'm just trying to reduce the number of fields we have, which impacts this problem), but I'd be happy to follow direction and start testing patches.

Just rerolling the patch.

Sorry, no time for additional profiling. We plan to get rid of metatag one day. Its not metatag's fault per se. Token processing is slow, and we have way too many fields. Its hard to know what fields are unused, after a site has been active for a few years.

Apparently there's also performance differences depending on how the tokens are used.
#3162152: Document performance gains from property tokens

patch no longer applies against the latest release. Is this patch now deprecated? I looked at the conflict it's pretty significant.

I could be wrong but i think this isn't needed anymore since metatag is roughly doing same logic now.

    if (!isset($this->processedTokenCache[$entity_identifier])) {
      $metatag_tags = $this->tagPluginManager->getDefinitions();
      foreach ($tags as $tag_name => $value) {
          ...
          $this->processedTokenCache[$entity_identifier][$tag_name] = $tag->multiple() ? explode(',', $value) : $value;
        }
    }

Yes, #2862747-16: Tokens to access individual meta tag values confirms that it basically took this patch and merged it into that issue over there.