Problem/Motivation

Hey!

Since token 1.17 introduced TokenModuleProvider (a
CacheCollector stored in cache.bootstrap), tokens
like [current-page:url] and [current-date:short]
can silently stop resolving after a token_info cache tag
invalidation, even though a full drush cr restores them.

Additionally, no hook_update_N was included in 1.17 to
invalidate the old token_info:{lang} cache key (renamed to
token_info_sorted:{lang}), so sites that did not run
drush cr on deploy hit this immediately.

Steps to reproduce

  1. Upgrade to token 1.17 without running drush cr, OR
  2. On a site with token 1.17 installed: save a webform or trigger any
    token_info tag invalidation.
  3. Observe that [current-page:url] and
    [current-date:short] render literally in webform emails
    instead of being replaced.
  4. Run drush cr — tokens work again.
  5. Trigger a token_info invalidation — tokens break
    again.

Proposed resolution

Override Token::resetInfo() to also call
TokenModuleProvider::reset() before shutdown. This clears
$storage and $keysToPersist so
CacheCollector::destruct() has nothing stale to write back.

Also add token_update_8117() to invalidate
token_info on upgrade, so sites that did not run
drush cr after the 1.17 deploy are automatically
remediated.

Remaining tasks

  • Add automated test covering the write-back scenario
  • User interface changes

    None.

    API changes

    Token::resetInfo() now also resets
    TokenModuleProvider's in-memory state as a side effect. Any
    caller that relied on TokenModuleProvider retaining its
    storage after resetInfo() will rebuild from cache on next
    access - which is the correct behaviour.

    Data model changes

    None.

    Comments

    max lovric created an issue. See original summary.

    max lovric’s picture

    Issue summary: View changes
    max lovric’s picture

    Issue summary: View changes
    max lovric’s picture

    Issue summary: View changes
    max lovric’s picture

    Title: TokenModuleProvider writes stale in-memory data back to cache.bootstrap after token_info tag invalidation » TokenModuleProvider writes stale in memory data back to cache.bootstrap after token_info tag invalidation
    berdir’s picture

    While I see that this information could in theory get out of sync, it's not really clear to me why that would happen in practice. Which module provides a token is generally not really something that changes during a request or at all.

    The steps to reproduce run drush cr. That could imply that _any_ cache is responsible for this. Can you already reproduce this when you _only_ clear this specific cache item?

    what exactly *is* the wrong information that is stored in the token module cache?

    berdir’s picture

    Status: Active » Postponed (maintainer needs more info)
    solide-echt’s picture

    Just to add some more info in trying to narrow down this issue:
    Similar issues with (then) APC caching and tokens have been reported way back, see https://www.drupal.org/project/token/issues/1782828. In my case I get errors of the type "AH01071: Got error 'PHP message: PHP Fatal error: Cannot redeclare token_token_info_alter()" or other functions under #[LegacyHook]. Strangely enough my server doesn't run APCU. Switching off Opcache doesn't make a difference too. The issue exists in both D10 and D11. Disabling Bigpipe, Internal Dynamic Page Cache and Internal Page cache does not resolve my issue. Drush cr idem.

    solide-echt’s picture

    As a follow up on my info: the issue seems to have been that PHP Composer did not remove token.tokens.inc when upgrading from token 1.16 to 1.17. To explain my workflow a bit more: I have a git repository containing only composer.json and composer.lock. After upgrading and commiting changes to git I deploy the updated composer files on a Plesk server and run PHP Composer Install function, ignoring the Update function in PHP Composer.
    So my advice to the TS is: check if token.tokens.inc is still present and if so, manually delete it.

    berdir’s picture

    Yes, you absolutely need to make sure that removed files are removed. composer definitely handles that, any change to a project completely replaces that. sounds like there is a problem with your workflow. I've seen similar issues from people in pathauto as well IIRC.

    Either way, it is not related to this issue at all.

    Also unrelated, especially to files and code, you should be using APCU, it has _significant_ impact on Drupal performance and is pretty much expected.