Token default values like [node:field_x]{100} break the expression when the token is resolved

Problem/Motivation

A token in an expression can have a default value, written right after it in curly braces. The Expression setting description documents this with the example [node:field_some_number]{100}: if the token cannot be resolved, 100 is used instead.

The default value works when the token cannot be resolved, but breaks the expression when the token is resolved, which is the usual case. The {100} wrapper is left in the expression:

[node:field_price]{100} * 2   becomes   1234.50{100} * 2

EvalMath cannot parse this. With Suppress Errors on (the default) the field is silently saved as 0 for the 2.x field types, and as an empty value for the legacy field_expression type.

The cause is in evaluateExpression(), in both ExpressionItemTrait and ExpressionItem. The line that removes the {...} wrappers is inside the loop over the tokens that are still unresolved after token replacement:

$remaining_tokens = $token_service->scan($expression);
foreach ($remaining_tokens as $tokens) {
  foreach ($tokens as $token) {
    ...
    // Clean up any remaining default value wrappers.
    $expression = preg_replace('/\{.*?\}/', '', $expression);
  }
}

When every token resolves, the loop never runs and the wrappers stay. The expression only works if some other token in it is still unresolved, which makes the behavior look random:

The code has been like this since the first commit (2017 for ExpressionItem, 2021 for ExpressionItemTrait), so all releases are affected.

Steps to reproduce

1. Add a Number (decimal) field field_price to a content type.
2. Add an Expression (decimal) field to the same content type, with the expression [node:field_price]{100} * 2.
3. Create a node without a price and save it. The expression field is 200, as expected.
4. Edit the node, set the price to 1234.5 and save it. The expression field is 0 instead of 2469.

Turning on Debug Mode (requires Devel) shows the token replaced expression as 1234.50{100} * 2.

Tested on 2.x-dev with Drupal 11 and Token 8.x-1.17. The :value token ([node:field_price:value]{100}) and the legacy field_expression type behave the same way.

Proposed resolution

Move the wrapper cleanup out of the loop, so that it runs after the default values are applied, whether or not any token is left unresolved. In both ExpressionItemTrait::evaluateExpression() and ExpressionItem::evaluateExpression():

    foreach ($remaining_tokens as $tokens) {
      foreach ($tokens as $token) {
        ...
      }
    }

    // Clean up any remaining default value wrappers.
    $expression = preg_replace('/\{.*?\}/', '', $expression);

The default values themselves are still applied inside the loop, so unresolved tokens keep working as before. Curly braces have no meaning in EvalMath, so removing them unconditionally cannot break a valid expression.

Tested with this change on both the 2.x field types and the legacy type:

Remaining tasks

• Merge request with the fix in ExpressionItemTrait and ExpressionItem.
• Review.

User interface changes

None.

API changes

None.

Data model changes

None. Existing entities keep their stored value until they are saved again (or, with Always Evaluate Expression? on, until they are rendered).