Need to figure out how to create nested tokens from the array token type

Marking as a stable release blocker.

Also see #1058856-3: Entity tokens not created for multi-value fields which contains a working-prototype (at least at the I time I did it) for that problem. I've used the entity API notation of list<taxonomy_term> to describe the data-type, what allows us to easily pass through the context of "the list items are terms" through.

The approach outlined in #1 seems to require re-implementing the "how is type X printed by default" logic, what shouldn't be required. That's already implemented and should be re-used.

yep, I guess all you'd need would be re-implementing the following function (or do something similar):

/**
 * Extracts the contained type for a list type string like list<date>.
 *
 * @return
 *   The contained type or FALSE, if the given type string is no list.
 */
function entity_property_list_extract_type($type) {
  if (strpos($type, 'list<') === 0 && $type[strlen($type)-1] == '>') {
    return substr($type, 5, -1);
  }
  return FALSE;
}

So the data-type can be extracted and passed on for further token replacements.

Subscribing.

Could some explain, and please forgive my ignorance, why do nested tokens need to be figured out for non-nested field tokens to work? Is it not overkill to build a whole object layer around nested/array tokens just to expose 1-dimensional fields?

Thanks in advance for explaining.

multivalue fields

There were multi-value fields in Drupal 6 and field tokens worked there. Maybe they only worked for the first value, so that's a limitation. But given the choice between no field tokens at all, or field tokens working the way they used to work, isn't it worth pursuing the latter first, then trying to deal with the harder one?

subscribe

Subscribing

sub

Are we planning on anything more than 'first' and 'last', plus delta values? I assume only those two 'named' deltas would be allowed.

We need to think about allowing for various options for joining multiple values. For instance, in a path we might want the tids to be joined using '+' instead of ',' and other places we might want to use a pipe or a space between them. So maybe 'keys' and 'join' need modifiers that define what would be used to join those values? Something like [array:values:join:?].

I would like to propose 'list<TYPE>' types (arrays) as follows: if [node:comments] is of type 'list<comment>', then [node:comments:first] and [node:comments:value:0] would be the first comment of the node, of type 'comment', of course. [node:comments:first:url] would then be the URL of the comment.

In hook_token_info_alter() we can scan the list of tokens for types of the form 'list<TYPE>' where TYPE is one of the 'needs-data' types, and then dynamically add those types with 'needs-data' => 'list<TYPE>' and having the usual array members provide 'TYPE'.

To avoid overloading the tree we suppress the 'list<TYPE>' types (list<node> includes 8 node subtrees!).

A bit of terminology: 'first', 'last', and 'value:?' are of the same type, so they should all either be called "element" or "value". I haven't found a way to make 'value:?' expandable though.

The attached patch implements the tree and the token replacement.

Here's an updated, re-rolled patch.

It makes 'value:?' work correctly (based on ID) and adds 'index:?' (zero-based index), which is much more convenient for enumerating elements.

Here's an updated, re-rolled patch.

I also added proper explanations for 'value:?' and 'index:?' to the tree.

(Keep in mind that this is not visible until some module actually exposes a 'list<TYPE>' type.)

Subscribe.

We need to have the element type in the name of the container type. So you prefer 'array' over 'list', right? That's fine with me, I just want to make sure I understand your request and to give you my thoughts for consideration:
— TYPE may not have a string representation. Typically, users will have to write something like [subs:nodes:0:title] rather than just [subs:nodes:0]. Since array is declared as "array of strings" it might make sense to use something other than 'array', to warn users about this.
— If entities and Rules already use 'list', it might make sense to use that.

Should we actually allow [subs:nodes:0] to return a node or should we use something like @strval() to try to cast it to a string? The returned object may or may not have a __toString() method...

One other question: we have [array:value:?] where ? is the key. Having to iterate over the keys in order to iterate over the values is quite cumbersome. That's why I'm proposing to add [array:index:?] where ? is the zero-based numeric index into the values. Do you still like [array:value:?] or would it make sense to add [array:key:?] and deprecate [array:value:?]? I think 'key' vs. 'index' would make a clearer distinction.

And a last one: do you agree that we should not try to show array subtrees in the tree? Trying to include an array caused all sorts of trouble for me...

1. The information about the element type is not readily available in hook_token() if it's not encoded in the type, such as 'array<TYPE>'. I don't think we want to require the hook_token() implementations to call the uncached token_info() and parse the token path that lead to the actual call. Besides, this is not possible as soon as you have a dynamic element in the token path.

Example: [subs:nodes:value:3:comments:value:2:title] would return the title of the third comment to the fourth node in a set of nodes. If we allow [subs:nodes:value:3:comments] to pass on an 'array<comment>', then we're on solid ground, but if we pass on only an 'array', then the tokens.token() will have to look up the 'array type' of [subs:nodes], which is quite fragile.

Taking this further, if we have [subs:entities:value:3:some_field] then we may not know the type of the entity nor the type of the field nor even whether it's multi-valued or not, but in hook_tokens() we can figure this out and pass on, say, and 'array<url>' if that's what it is, so that [subs:entities:value:3:some_field:first:path] can work. The user may know what it is because he knows how the entities were selected, and the entity knows it, but token_info() doesn't.

For Subscriptions I've implemented [subs:files:field_FILES:value:?] to be an 'array<file>'. I don't know what name the admin will give to the field that should hold file attachments, but when I see 'files' I can interpret the field_FILES component to be the name of a multi-value file field. If he calls his field 'attachments', then Subscriptions can resolve [subs:files:attachments:value:count] to be the number of attachments, even though 'attachments' is completely unknown to token_info().

3.

The point of having value:? based on key is because the array keys could be anything. For fields they always start with 0. But user roles, for example do not. If people want the first value, they can use [array:first].

I agree with that, but getting the second value means resolving [array:values:[array:keys:1]]. Iterating over all values means getting [array:count], then getting [array:keys:$i] for each $i in 0..$count-1 to retrieve all the $keys, and finally with each $key to get [array:values:$key]...

I certainly want to keep the key-based lookup, it is definitely useful, but I want to add a position-based lookup, such as [array:index:1] or [array:index:$i]. Can you consent to that?

Now, if we have [array:values:$key] and [array:index:$i] both returning "values", then the naming scheme is not so clear anymore. That's why I'm proposing to rename "values" to "key". That way the x in [array:x:y] will say what y is, either the 'key' or the (positional) 'index'.

4.

And yes, we need to display the subtrees in context in the UI otherwise people will not know how to use them.

If we have an array of nodes, then 'first', 'last', 'values:?' and 'index:?' will have an entire node subtree. 'reverse' will add another set of 'first', 'last', 'values:?' and 'index:?', giving us eight (!) node subtrees just by adding a single 'nodes' token. Having a couple of those in the tree breaks my PHP interpreter (Apache crash) and my Firefox...

I've just published Subscriptions ALPHA5 and Mail Editor ALPHA2, which use the 'list<TYPE>' tokens (based on Token 7.x-1.0-beta6 plus the patch in #20). This serves as a proof of concept.

I'm attaching a screenshot of part of the token tree.

Where is that cache?

And, did you read on? Even if Token caches that information, what it can cache is static information that's available before token_tokens() starts, or am I missing something? I don't see how Subscriptions could inject type information into Token that only becomes available during hook_tokens() execution — I'll need help there...

Any comments to #4?

How can we proceed here? This is blocking the development of Subscriptions.

If we cannot discuss this here and come up with a viable solution in Token then I have to implement it elsewhere.

...coming from #1280326: Add field level token integration.

Would it be reasonable to add a hook that might allow the number of named deltas to be customized?

#1334456: Enable ajax on-demand loading of sub-trees in the Token help would also be useful.

Subscribe

Tom, see https://www.drupal.org/node/1306444

Unassigned @Dave Reid

Eight years later, after a lot happened, i wonder if we want to add tokens that leverage GraphQL.

That still doesn't exist in the 8.x version.
What if we had a really simple implementation allowing to use NestedArray methods to get a deep value?
Patch to come.

Here is a really simple implementation.

@Maintainers: Who can decide how to proceed here? And especially review the MR!26 from @DuaelFr so we can have a plan? :)