Updating to Token 1.13 breaks smart_date range tokens

We're seeing the same behavior using Drupal 10.1.5 and the latest version of Token Filter and Smart Date. Reverting to Token 1.12 fixed the issue for us (for now).

FYI #3397358: Menu link submit save callback detection breaks some AJAX forms was opened as another regression in the new release.

I just ran into this myself, and can confirm that reverting Token to 1.12 got my tokens working again.

Update: Although we thought simply reverting to Token 1.12 fixed our issue with smart_date tokens, we're seeing continued odd behavior in a subset of cases.

The smart_date tokens ARE available for our needs as automatic entity labels, but since we're using a complex combination of moderation_state and a multi-step form, we do have cases where a user might save our form as as "Not Ready" (a published state we created to replace "draft") - call it "Save 1" - BEFORE editing the fields in which we use smart_date. If they do this, then return to edit, adding in the smart_date values, the node gets saved again (call that "Save 2") WITHOUT the smart_date range token values being included in the node title.

Many of our users stop there (they have completed the form and are satisfied marking it as "Submitted" - another published moderation_state). We see that the smart_date tokens do NOT get added to/replaced in the title, however, UNTIL the next save ("Save 3") of that same node (even without further edits to any field).

It seems as though the automatic entity label is using the smart_date tokens from the original node field values (dates still empty) on "Save 2", but everything is fine and the tokens are replaced as expected on "Save 3".

In contrast, if the user comes in and fills out the form completely (new entity being inserted, smart_date field values added on first save), then the smart_date tokens are replaced correctly in the title of the node.

I don't know what could cause that, maybe the hook refactoring?

The problem is the same using a pattern with pathauto module :

Using the following pattern to generate automatic URL for my event pages :
events/[node:field_smart_date:value-custom:Y-m-d]_[node:title]
The result for "My first event" is
events/_my-first-event
instead of
events/2023-11-17_my-first-event

Same problem with value-format:my-id into the pattern.

Only with Drupal 10 (10.1.6 tested) ... OK with Drupal 9 (9.5.11 tested)
Smart-date v4.0.3 or v4.1.0-rc3 ... same problem.

I had to downgrade to token:1.12 to fix the problem on all my websites.

See issue #3402367: Token with "value-custom:?" return an empty string

Its occuring because the value set in the hook function smart_date_tokens is getting replaced by null value inside the hook function token_tokens. This is happening while invoking hook_tokens(). Tried updating the smart_date module weight to 10 and things worked fine even with token:1.13 module.

So we can either solve this issue by increasing other module's weight. Or we need to alter hook_tokens() calling in such a way that if some other module is setting a value, we should avoid replacing it with null value. My suggestion is the second method, as some other modules may also face this same issue in the future.

Yeah, that has to be a regression from #3088219: Inconsistent loading of third party hook implementations. Maybe this hook could be changed to not replace a value with null, assuming we actually have access to the values from the other hook?

Same issue here.
Will try to downgrade to 1.12 unless: @sijumpk Can you explain what you mean by updating weight to 10?

Its updating the smart_date module weight. Its done by adding hook_update_N to the smart_date.install file and running the datebase update script. Here is the hook script I added to the install file for accomplishing it.

/**
 * Increase smart_date module weight.
 */
function smart_date_update_8302() {
  module_set_weight('smart_date', 10);
}

Do _not_ add update functions to contrib modules. If the module adds its own 8302, that will not be executed on your site and your site might break.

Add it in a custom module, run it with drush scr/ev or change the weight in core.extension and do a config import of that.

Sure @Berdir its my mistake. Just provided the code for confirming it will work, created that in my local to check that solution. Thats why didn't created a patch out of it.

@Transmitter. You can also try the contributed module https://www.drupal.org/project/modules_weight for altering the module weight. Do remember to clear the cache after changing the weight. Before implementing it in production, better to check with @mandclu. I am not sure whether it will interfere with some other functionality of Smart Date module

@Berdir, is there any reason for inserting a null value to the field tokens (inside _field_tokens function)? Tried adding a condition to check the value for null before inserting it. By doing so we can avoid the value overwriting problem. Adding the patch for it.

I would expect that will result in some fails.

The problem is that not replacing a token if there is no value is that it will result in leftover tokens in those cases. I remember that being changed a while ago to not have that problem.

If we don't see replacements of other modules already in the list then I would expect we need to handle this in the Token class, and not overwrite values with nothing if there already is a value.

There is a merge request for #3433267: Smart date value custom format tokens empty that seems to fix this issue by moving Smart Date's token implementations to the end of the list. Any thoughts here on whether that's a better way to fix the problem?

/**
 * Implements hook_module_implements_alter().
 *
 * Move smart_date hook_tokens() implementation to the end of the list to avoid
 * conflict with the Tokens module's remplacement for custom date format/values.
 */
function smart_date_module_implements_alter(&$implementations, $hook) {
  switch ($hook) {
    case 'tokens':
      $group = $implementations['smart_date'];
      unset($implementations['smart_date']);
      $implementations['smart_date'] = $group;
      break;
  }
}

I ran into the same issue just now, and I can confirm that both approaches from #14 and #16 are solving the issue.

As to @Berdir's concern in #15 I don't think that applies. That _field_tokens() method only gets called, if the field module is enabled, and then it adds a token replacement with value NULL. If the field value weren't available, that NULL replacement wouldn't be added, just like adding the check from #14

FYI the proposed fix in #16 has been merged into Smart Date and is now included in the 4.1.0-rc7 release. Should we close this issue?

I think this can be closed as suggested above by @mandclu. Hence I am closing this issue. Feel free to reopen it if needed.

Thanks!