One step blocking Field tokens is we need to be able to create complex nested tokens using the array token type. If this can be solved independent of the Field token mess, like solving a [node:terms:first::tid] token we will be one step closer.

Files: 
CommentFileSizeAuthor
#26 Subscriptions_Token-Tree.png28.82 KBsalvis
#20 token.list-of-type.1195874.20.patch6.15 KBsalvis
PASSED: [[SimpleTest]]: [MySQL] 284 pass(es). View
#19 token.list-of-type.1195874.19.patch5.71 KBsalvis
PASSED: [[SimpleTest]]: [MySQL] 284 pass(es). View
#18 token.list-of-type.1195874.18.patch5.16 KBsalvis
PASSED: [[SimpleTest]]: [MySQL] 284 pass(es). View

Comments

Dave Reid’s picture

This is the current code I was playing with:

class TokenArrayObject extends ArrayObject {
  private $joinString = ', ';

  public function setJoinString($string) {
    $this->joinString = $string;
  }

  public function getStringValue($index) {
    return (string) $this[$index];
  }

  public function __toString() {
    $strings = array();
    foreach ($this as $key => $value) {
      $strings[] = $this->getStringValue($key);
    }
    return implode($this->joinString, $strings);
  }

  public function reverse() {
    $data = $this->getArrayCopy();
    $data = array_reverse($data, TRUE);
    $class = get_class($this); // This should extend the same type of object.
    return new $class($data);
  }

  public function keys() {
    $data = $this->getArrayCopy();
    $data = array_keys($data);
    return new TokenArrayObject($data);
  }
}

class TermTokenArrayObject extends TokenArrayObject {
  public function getStringValue($index) {
    return $this[$index]->name;
  }
}

class FileTokenArrayObject extends TokenArrayObject {
  public function getStringValue($index) {
    return $this[$index]->filename;
  }
}

Somehow we want our 'array' token type to use the getStringValue() or magic __toString() methods with its token values.

webchick’s picture

Marking as a stable release blocker.

fago’s picture

Also see #1058856-3: Entity tokens not created for multi-value fields which contains a working-prototype (at least at the I time I did it) for that problem. I've used the entity API notation of list<taxonomy_term> to describe the data-type, what allows us to easily pass through the context of "the list items are terms" through.

The approach outlined in #1 seems to require re-implementing the "how is type X printed by default" logic, what shouldn't be required. That's already implemented and should be re-used.

Dave Reid’s picture

We're not depending on entity API for Token, so we do have to re-implement it to some degree.

fago’s picture

yep, I guess all you'd need would be re-implementing the following function (or do something similar):

/**
 * Extracts the contained type for a list type string like list.
 *
 * @return
 *   The contained type or FALSE, if the given type string is no list.
 */
function entity_property_list_extract_type($type) {
  if (strpos($type, 'list<') === 0 && $type[strlen($type)-1] == '>') {
    return substr($type, 5, -1);
  }
  return FALSE;
}

So the data-type can be extracted and passed on for further token replacements.

MustangGB’s picture

adamdicarlo’s picture

Subscribing.

thebuckst0p’s picture

Could some explain, and please forgive my ignorance, why do nested tokens need to be figured out for non-nested field tokens to work? Is it not overkill to build a whole object layer around nested/array tokens just to expose 1-dimensional fields?

Thanks in advance for explaining.

jastraat’s picture

multivalue fields

thebuckst0p’s picture

There were multi-value fields in Drupal 6 and field tokens worked there. Maybe they only worked for the first value, so that's a limitation. But given the choice between no field tokens at all, or field tokens working the way they used to work, isn't it worth pursuing the latter first, then trying to deal with the harder one?

zilverdistel’s picture

subscribe

BenK’s picture

Subscribing

Dave Reid’s picture

Thinking that something using #1062498: Add a 'default token' information to token types might also have to be involved.

$terms = array(
  // Array of term objects.
);
$default_term_token = 'name';
$tokens = token_generate('term', array($default_term_token => '[node:terms:first]'), array('term' => $terms[0]));
Dave Reid’s picture

Basically:

Given an array like this:

array (
  1 => 
  stdClass::__set_state(array(
     'tid' => '1',
     'vid' => '1',
     'name' => 'term1',
     'description' => '',
     'format' => 'filtered_html',
     'weight' => '0',
     'vocabulary_machine_name' => 'tags',
  )),
  2 => 
  stdClass::__set_state(array(
     'tid' => '2',
     'vid' => '1',
     'name' => 'term2',
     'description' => '',
     'format' => 'filtered_html',
     'weight' => '0',
     'vocabulary_machine_name' => 'tags',
  )),
  3 => 
  stdClass::__set_state(array(
     'tid' => '3',
     'vid' => '1',
     'name' => 'test3',
     'description' => NULL,
     'format' => NULL,
     'weight' => '0',
     'vocabulary_machine_name' => 'tags',
  )),
)

We need to be able to have something like this:
token_replace('...', array('array' => $array));

Perform the following replacements as well as be able to show itself as a 'term' token in the UI so that a user can drill-down to the term-specific tokens.

[array:first] = term1
[array:first:tid] = 1
[array:first:vocabulary] = Tags
[array:values:0] = term1
[array:values:0:tid] = 1
[array:values:0:vocabulary] = Tags
[array:values:1] = term2
[array:values:1:tid] = 2
[array:values:1:vocabulary] = Tags
[array:values:2] = term3
[array:values:2:tid] = 3
[array:values:2:vocabulary] = Tags
[array:values:last] = term3
[array:values:last:tid] = term3
[array:values:last:vocabulary] = Tags
[array:values:keys] = 1, 2, 3
[array:values:join] = term1, term2, term3
bendiy’s picture

sub

KarenS’s picture

Are we planning on anything more than 'first' and 'last', plus delta values? I assume only those two 'named' deltas would be allowed.

We need to think about allowing for various options for joining multiple values. For instance, in a path we might want the tids to be joined using '+' instead of ',' and other places we might want to use a pipe or a space between them. So maybe 'keys' and 'join' need modifiers that define what would be used to join those values? Something like [array:values:join:?].

Dave Reid’s picture

We'd have:

[array:first:*]
[array:last:*]
[array:value:N:*] - dynamic token

We already have [array:join:CUSTOM-TEXT] so for a term reference field people could use [node:field_terms:keys:join:+].

salvis’s picture

FileSize
5.16 KB
PASSED: [[SimpleTest]]: [MySQL] 284 pass(es). View

I would like to propose 'list<TYPE>' types (arrays) as follows: if [node:comments] is of type 'list<comment>', then [node:comments:first] and [node:comments:value:0] would be the first comment of the node, of type 'comment', of course. [node:comments:first:url] would then be the URL of the comment.

In hook_token_info_alter() we can scan the list of tokens for types of the form 'list<TYPE>' where TYPE is one of the 'needs-data' types, and then dynamically add those types with 'needs-data' => 'list<TYPE>' and having the usual array members provide 'TYPE'.

To avoid overloading the tree we suppress the 'list<TYPE>' types (list<node> includes 8 node subtrees!).

A bit of terminology: 'first', 'last', and 'value:?' are of the same type, so they should all either be called "element" or "value". I haven't found a way to make 'value:?' expandable though.

The attached patch implements the tree and the token replacement.

salvis’s picture

Status: Active » Needs review
FileSize
5.71 KB
PASSED: [[SimpleTest]]: [MySQL] 284 pass(es). View

Here's an updated, re-rolled patch.

It makes 'value:?' work correctly (based on ID) and adds 'index:?' (zero-based index), which is much more convenient for enumerating elements.

salvis’s picture

FileSize
6.15 KB
PASSED: [[SimpleTest]]: [MySQL] 284 pass(es). View

Here's an updated, re-rolled patch.

I also added proper explanations for 'value:?' and 'index:?' to the tree.

(Keep in mind that this is not visible until some module actually exposes a 'list<TYPE>' type.)

Niklas Fiekas’s picture

Subscribe.

Dave Reid’s picture

@salvis: Could you please merge this with the work already done with the 'array' token type? I'd rather not have two different types.

salvis’s picture

We need to have the element type in the name of the container type. So you prefer 'array' over 'list', right? That's fine with me, I just want to make sure I understand your request and to give you my thoughts for consideration:
— TYPE may not have a string representation. Typically, users will have to write something like [subs:nodes:0:title] rather than just [subs:nodes:0]. Since array is declared as "array of strings" it might make sense to use something other than 'array', to warn users about this.
— If entities and Rules already use 'list', it might make sense to use that.

Should we actually allow [subs:nodes:0] to return a node or should we use something like @strval() to try to cast it to a string? The returned object may or may not have a __toString() method...

One other question: we have [array:value:?] where ? is the key. Having to iterate over the keys in order to iterate over the values is quite cumbersome. That's why I'm proposing to add [array:index:?] where ? is the zero-based numeric index into the values. Do you still like [array:value:?] or would it make sense to add [array:key:?] and deprecate [array:value:?]? I think 'key' vs. 'index' would make a clearer distinction.

And a last one: do you agree that we should not try to show array subtrees in the tree? Trying to include an array caused all sorts of trouble for me...

Dave Reid’s picture

I don't think its necessary to create a whole other type. We can provide that information in hook_token_info() itself:

  $info['tokens']['term']['parents'] = array(
    'name' => t('Parents'),
    'description' => t('All the parents of the term, start with the root term, ending with the parent term.'),
    'type' => 'array',
    'array type' => 'term',
  );

The point of having value:? based on key is because the array keys could be anything. For fields they always start with 0. But user roles, for example do not. If people want the first value, they can use [array:first].

And yes, we need to display the subtrees in context in the UI otherwise people will not know how to use them.

salvis’s picture

1. The information about the element type is not readily available in hook_token() if it's not encoded in the type, such as 'array<TYPE>'. I don't think we want to require the hook_token() implementations to call the uncached token_info() and parse the token path that lead to the actual call. Besides, this is not possible as soon as you have a dynamic element in the token path.

Example: [subs:nodes:value:3:comments:value:2:title] would return the title of the third comment to the fourth node in a set of nodes. If we allow [subs:nodes:value:3:comments] to pass on an 'array<comment>', then we're on solid ground, but if we pass on only an 'array', then the tokens.token() will have to look up the 'array type' of [subs:nodes], which is quite fragile.

Taking this further, if we have [subs:entities:value:3:some_field] then we may not know the type of the entity nor the type of the field nor even whether it's multi-valued or not, but in hook_tokens() we can figure this out and pass on, say, and 'array<url>' if that's what it is, so that [subs:entities:value:3:some_field:first:path] can work. The user may know what it is because he knows how the entities were selected, and the entity knows it, but token_info() doesn't.

For Subscriptions I've implemented [subs:files:field_FILES:value:?] to be an 'array<file>'. I don't know what name the admin will give to the field that should hold file attachments, but when I see 'files' I can interpret the field_FILES component to be the name of a multi-value file field. If he calls his field 'attachments', then Subscriptions can resolve [subs:files:attachments:value:count] to be the number of attachments, even though 'attachments' is completely unknown to token_info().

3.

The point of having value:? based on key is because the array keys could be anything. For fields they always start with 0. But user roles, for example do not. If people want the first value, they can use [array:first].

I agree with that, but getting the second value means resolving [array:values:[array:keys:1]]. Iterating over all values means getting [array:count], then getting [array:keys:$i] for each $i in 0..$count-1 to retrieve all the $keys, and finally with each $key to get [array:values:$key]...

I certainly want to keep the key-based lookup, it is definitely useful, but I want to add a position-based lookup, such as [array:index:1] or [array:index:$i]. Can you consent to that?

Now, if we have [array:values:$key] and [array:index:$i] both returning "values", then the naming scheme is not so clear anymore. That's why I'm proposing to rename "values" to "key". That way the x in [array:x:y] will say what y is, either the 'key' or the (positional) 'index'.

4.

And yes, we need to display the subtrees in context in the UI otherwise people will not know how to use them.

If we have an array of nodes, then 'first', 'last', 'values:?' and 'index:?' will have an entire node subtree. 'reverse' will add another set of 'first', 'last', 'values:?' and 'index:?', giving us eight (!) node subtrees just by adding a single 'nodes' token. Having a couple of those in the tree breaks my PHP interpreter (Apache crash) and my Firefox...

salvis’s picture

FileSize
28.82 KB

I've just published Subscriptions ALPHA5 and Mail Editor ALPHA2, which use the 'list<TYPE>' tokens (based on Token 7.x-1.0-beta6 plus the patch in #20). This serves as a proof of concept.

I'm attaching a screenshot of part of the token tree.

Dave Reid’s picture

The information about the element type is not readily available in hook_token() if it's not encoded in the type, such as 'array'. I don't think we want to require the hook_token() implementations to call the uncached token_info() and parse the token path that lead to the actual call. Besides, this is not possible as soon as you have a dynamic element in the token path.

Example: [subs:nodes:value:3:comments:value:2:title] would return the title of the third comment to the fourth node in a set of nodes. If we allow [subs:nodes:value:3:comments] to pass on an 'array', then we're on solid ground, but if we pass on only an 'array', then the tokens.token() will have to look up the 'array type' of [subs:nodes], which is quite fragile.

We cache that information in token module so that is not a problem. Once we hit an array token we also extract the value type and pass it along in either $data or $options in token_generate().

salvis’s picture

Where is that cache?

And, did you read on? Even if Token caches that information, what it can cache is static information that's available before token_tokens() starts, or am I missing something? I don't see how Subscriptions could inject type information into Token that only becomes available during hook_tokens() execution — I'll need help there...

Any comments to #4?

salvis’s picture

How can we proceed here? This is blocking the development of Subscriptions.

If we cannot discuss this here and come up with a viable solution in Token then I have to implement it elsewhere.

Dave Reid’s picture

Priority: Normal » Major
klonos’s picture

Dave Reid’s picture

Since I'm not going to be changing how the root-level field tokens work and we have not yet implemented nested field tokens, for which this issue blocked, I've rolled a 7.x-1.0 release without this issue. I hope to work on this for a 7.x-1.1 release.

DamienMcKenna’s picture

Issue summary: View changes

Would it be reasonable to add a hook that might allow the number of named deltas to be customized?

DamienMcKenna’s picture

Tom Dowling’s picture

Subscribe

anavarre’s picture