Problem/Motivation

We have the ability to do token replacements in Core. But we don't have a token browser. It's currently provided by the Token contrib module (http://drupal.org/project/token). Without a token browser, there is no way for users setting up tokens to discover what tokens are available to them.

Proposed resolution

Add something similar to the browser from the contrib Token module to core.

Design considerations

  • Efficient (code/load time - current treeTable implementation of the contrib Token 7.x module breaks with lots of tokens)
  • Accessible (current TreeTable implementation is not!)
  • Usable
    • Efficient to find the token you are looking for ("scannable")
    • Search capability - which may or may not be different from autocomplete
    • Hierarchical structure of tokens reflected in hierarchical-looking UI (intuitive)
    • Screen real estate (keeping mobile in mind too!)
    • Maybe different modes depending on the scope of the token list?
      - Flat list if there are only a few
      - Tree list and/or hierarchical autocomplete and/or search if the list is longer
      - Not loading the entire list initially if it's really long (page load time)
  • Reusable UI pattern that can be used in Rules, Panels, Blocks, Layouts, Views, Etc. for both tokens and "token-like things".
  • Naming conventions / standards for "token-like things"

Scope for this issue

This issue will be limited to building a tree-based token browser.

We'll spin off other issues as necessary to deal with:
- Autocomplete
- Search
- Token naming
- A UI that changes based on the size of the list (flat -> tree etc.)

Remaining tasks

1. Make a patch that adds the browser similar to the contrib Token module to core. However, it shouldn't actually *be* that browser, since it's not efficient/scalable and is also not accessible.
2. Make sure there are tests for it.
3. Make sure it is accessible.
4. Make sure it is usable.

User interface changes

Core would have a token browser, which would be displayed whenever tokens are usable in the admin UI.

API changes

None.

Original report by eaton

Many different parts of the Drupal admin interface use placeholder tokens inside of a longer stretch of text to represent "some value that will be replaced later." Sending out welcome emails to users is a common case: core allows admins to enter the text of the mail, with placeholder tokens like %user-name and %site-url.

Token module is a popular contrib module that offers similar functionality to modules that need placeholders/wildcards for other purposes. Modules can provide new tokens, like "[og-name]" for "The name of the organic group that a node belongs to" or [site-slogan] for "The site slogan the admin entered on the site settings page. Users can then use those placeholder tokens in any token-enabled field (like pathauto's configuration page). Token module is being proposed for inclusion in Drupal 7 core, because its centralized 'registry' of placeholder tokens is useful all over core and contrib.

The challenge (and it's a tricky one) is how to properly explain to users what tokens are available to be used and what they mean when they are presented with a field that can "accept" tokens. Several solutions to this exist right now, and some brainstorming has been going on for the Pathauto module, but ironing it out for core would be very beneficial.

Right now, in Drupal 6, token module provides a helper function that just spits out a table of all the tokens that are available. This is less than ideal, because with a number of token-providing modules installed, the list can get really really long. Some modules (like pathauto) manually build their own lists of tokens broken up by the 'kind' of token. (Node related tokens, user related tokens, etc.)

Some existing links include:
http://groups.drupal.org/node/19500
http://www.flickr.com/photos/mverbaar/3595675392/in/set-72157619245872526/

What are the essential 'bits of information' that might be conveyed for tokens?

1) The 'token type' -- node, user, etc.
2) The token as it is used by a module -- for example 'title' or 'uid' or 'mail'
3) The user-friendly name of the token -- for example "Node title" or "User ID"
4) An optional extended description of the token -- for example, a fully formatted date demonstrating how the 'date' token will appear
5) The token as it can be entered by a user -- for example [user:name]
6) The token as it can be entered if chained with another token -- for example [node:author:name] instead of [user:name]

In the quick omnigraffle mockup below, I've listed an example of the sort of things that might be commonly entered into token-enabled fields. The fields themselves don't have to do anything special with the tokens -- they just need to have some way of presenting the 'possible' tokens. Some existing ideas include: a collapsible browsable list of tokens, grouped by token type (like 'node' and 'user' and 'site'). Auto-complete in the text fields themselves. A popup. Something made of magic.

Any thoughts? Anyone? Beuller?

Files: 
CommentFileSizeAuthor
#112 drupal_514990_112.patch17.41 KBXano
PASSED: [[SimpleTest]]: [MySQL] 59,319 pass(es). View
#79 tokenbrowser.png129.48 KBjhodgdon
#77 token-browser-proof-concept-v2.patch35.89 KBjhodgdon
PASSED: [[SimpleTest]]: [MySQL] 49,020 pass(es). View
#74 token-browser-proof-concept.patch31.99 KBjhodgdon
FAILED: [[SimpleTest]]: [MySQL] 49,004 pass(es), 1 fail(s), and 0 exception(s). View
#49 514990-token-browser-49.patch36.37 KBjhodgdon
PASSED: [[SimpleTest]]: [MySQL] 48,835 pass(es). View
#48 514990-token-browser-48.patch33.78 KBjhodgdon
PASSED: [[SimpleTest]]: [MySQL] 48,883 pass(es). View
#48 arrow-down.png118 bytesjhodgdon
#48 arrow-right.png127 bytesjhodgdon
#43 Mockup.png30.94 KBGaelan
#28 treetablenojavascript.png102.79 KBRobLoach
#25 mysql.drupal7dev.local - Mozilla Firefox_003.png65.77 KBDave Reid
#20 tokens_UI.jpg86.49 KBarhak
#7 token-mockups-take9.png157.38 KBeaton
#5 tokens.jpg31.06 KBBojhan
#4 token-mockup.png44.09 KBeaton
tokens.png24.99 KBeaton

Comments

eaton’s picture

Title: d7ux microproject: » d7ux microproject: User interface for browsing available tokens?

Whoops. Should have titled this properly.

Also, the existing issue for adding tokens to core lives at #113614: Add centralized token/placeholder substitution to core

Bojhan’s picture

Placeholder for an actual comment. Or Subscribing

Dave Reid’s picture

Popup modal dialog FTW.

eaton’s picture

FileSize
44.09 KB

After chatting with Bojhan, here's one proposal for how tokens could be presented in the UI...

Bojhan’s picture

FileSize
31.06 KB

tokens.jpg

So a update, I think one problem I did find is that - with descriptions (which is fine), is this interaction is unprecedented - so one behavior we should try to mimic is that of collapsing. I see a certain problem, with beign able to drill down but not beign able to move up quickly, so I think exposing the other tokens might be valuable - but this is by no means a set opinion.

kika’s picture

So it's basically a generic combined tree/table control? How many levels we need? Can we do it based on existing fieldsets http://groups.drupal.org/node/19500#comment-76033 or should we create a new UI element?

eaton’s picture

FileSize
157.38 KB

kika, the tricky parts are:

  1. There's a variable (and potentially very deep) number of levels, depending on what kinds of things a user is making tokens for
  2. The list of tokens for each level has the potential to be very long if modules add a lot of tokens (nodes in particular)
  3. The full 'depth' of the tree cannot be pre-rendered, because it can recurse.

Together, those pieces make it a bit of a doosie. The following rough mockup also reflects some of the stuff Bojhan and I were discussing...

kika’s picture

- Correct, this is hybrid "tree/expandable table" control then. It would be nice to have it as genric UI element.
- what are the proposed actions user can to with this widget? Perhaps I am mis-reading the mocks but the tokens 1.0-style guidance "here are the codes you can insert into a text field: [author-name], [node-title] ..." are lost in the new UI. How the user should know he needs to mash together active tree elements [node:author:last-login:short] in order to get the preferred token code? Or is there a planned JS trick to automatically paste the complete token code to the textfield from the active row?

eaton’s picture

Or is there a planned JS trick to automatically paste the complete token code to the textfield from the active row?

That's what I was thinking.

If we want to take a brutal shortcut route, we can just ignore any token 'chaining' entirely and only show top-level tokens, using a display style that mirrors token 1.x. ultimately, something like this is necessary to support the chaining...

sun’s picture

sun’s picture

Issue tags: +API clean-up

Introducing a new tag for feature freeze: API clean-up.

SeanBannister’s picture

Yeah the mockup in #7 is heading down the right path. If we take a shortcut and don't provide a way to see chaining it will drastically decrease the number of users who will realize this new ability exists.

Deciphered’s picture

Wondering if there is any movement on this? / subscribing.

dmitrig01’s picture

patcon’s picture

Why not do it like Facebook's @mention UI? No list below the text field at all -- just start typing and it autosuggests.

heyrocker’s picture

subscribe

yoroy’s picture

Version: 7.x-dev » 8.x-dev

Moving on up.

Dave Reid’s picture

Component: other » token system
arhak’s picture

Component: token system » base system

@#8 "Or is there a planned JS trick to automatically paste [...] to the textfield [...]?"
@#9 "That's what I was thinking."

two points:
1- downgrade gracefully when JS is disabled (therefore design without entirely relying on JS: autocompletion, onclick events, etc)
2- if a page with 10-15 textfields/textareas provided by the same module wishes to show up only ONE table of tokens (since it might be huge):
2a-- to which textfield/textarea will be the token pasted?
2b-- forced to use one huge table for each tiny textfield?!

another important point:
3- Note that collapsible fieldsets lost their collapsibility within vertical tabs (and BTW lost their titles as well)

I truly believe that the better approach should be find out a logical (human readable) way to display that bunch of data
I also believe that it can be implemented in contrib (kind of token_ui or token_helper module) AND IF it is ready before D7 release it might even get into core (a couple of theme functions won't hurt anyone IMO, otherwise long live in contrib apply for D8)

arhak’s picture

FileSize
86.49 KB

Currently I have no solution for this
but this was my first shot

the algorithm to build this table was:
1- in this case the consumer module only matters about [comment:*] and [site:*] tokens (the old D6 scopes for 'comment' and 'global')
2- fetch all the tokens
3- pick selected types and bundle them in an array ready for theme('table', ...)
4- for each one see what chains they have and take note of the type and the key of the chain (i.e. for [comment:parent:*] take note that the chain is 'parent' but its type is 'comment')
5- pick those chains and bundle them in another array ready for theme('table', ...)
6- for the current chains at hand repeat 4-5-6 until all chain's keys have been marked as considered
7- output the table of 'Tokens' with token like this [comment:cid], [comment:parent:*] to remark what is a chain an what's not (also prepend to chain descriptions the bold word 'chains')
8- output the table of 'Chains' with chain fragments like this *:parent:cid, *:node:created:*

the result is that chains that appeared first are nearest the top while, lets say for instance, date chains appear bellow node chains (depending on the case they might be even, but it is just an example)

Dave Reid’s picture

I think the planned jQuery UI 'Tree' element would be good for us here. :)

Dave Reid’s picture

#735354: User token types should *not* be nested is a big blocker for any kind of token UI or browser so it could use a quick review.

arhak’s picture

temporary you may use the type name as the group name, e.g "Comment tokens", "User tokens", and would have "Current-user tokens"

that's how I didn't hit that one

  foreach ($token_list as $type => $tokens) {
    $rows[] = array(array('data' => drupal_ucfirst($type) .' '. t('tokens'), 'class' => 'region', 'colspan' => 3));

so, it is not so blocker that issue, you may work and enhance the section header later on

Dave Reid’s picture

Issue tags: -token, -D7UX, -API clean-up

Here's a *working* token UI I'm developing for Token D7. Work in progress of course, but feedback is welcome. :)

Dave Reid’s picture

Here's a *working* token UI I'm developing for Token D7. Work in progress of course, but feedback is welcome. :)

EDIT: Thanks Guru Mediation. :P

Dave Reid’s picture

Issue tags: +token, +D7UX, +API clean-up

Live demo up at http://www.davereid.net/d7/examples/token, will post a token issue patch shortly. It's currently blocked since there isn't any good jQuery tree/table plugin that is also licensed with GPLv2. I contacted the author of the one I liked best to see if he would dual-license MIT/GPLv2.

SeanBannister’s picture

Really like this, it'd look great with a slide effect similar to the default way we display fieldsets.

RobLoach’s picture

FileSize
102.79 KB

I really like how TreeTable looks and functions. As an added bonus, it's actually not that bad when JavaScript isn't available too. Can't click on the text to copy in the token, obviously.

I looked for a similar solution that uses jQuery UI and found jQuery UI Tree, which unfortunately doesn't look as nice. But, it would be nice to have something that is future proof through the upgrade to jQuery 1.4 and jQuery UI 1.8.

..... What does SimpleTest use? We have something similar to this effect there, don't we?

eaton’s picture

Dave Reid is my hero.

I think there's a lot of polishing that can be done with the UX here, but this kind of browser interface is a huge boost.

Bojhan’s picture

Ehh, how is it diffrent from what I purposed in #5? Eitherway, this is not Drupal 7 stuff obviously.

RobLoach’s picture

Bojhan: The difference is that Dave made an implementation of it.

Dave Reid’s picture

Got a response from Ludo, the author of the treeTable jQuery plugin and he's open to dual licensing his plugin with MIT/GPLv2.

Frando’s picture

By the way, you're free to relicense MIT to GPLv2 yourself, MIT is fully compatible with the GPL and specifically allows you to sublicense work covered by MIT (IANAL).

Great work! The tree table looks very smooth in the demo and perfectly suited.

Dave Reid’s picture

Didn't know that, but I'd also feel much better just having the authors permission and approval, plus its fun to let someone know that you really like and appreciate their work. :)

Bojhan’s picture

@Rob Loach : Well if I could, I would :). Eitherway good work, I do think putting the arrows inside the table will make more sense.

Dave Reid’s picture

Component: base system » token system

I've made an official patch against the Token module for D7 and it wouldn't be too hard to do the same against core, although I want to know for sure before I put effort into that.

Follow #738372: Token UI browser + jQuery treeTable for all the patchy token UI goodness.

Dave Reid’s picture

Title: d7ux microproject: User interface for browsing available tokens? » Add a token browser UI for core
Issue tags: -token, -D7UX, -API clean-up

This is no longer a microproject or related to D7UX anymore.

tedbow’s picture

I am interested in taking this on. Would this mean moving Token module into core?
Or would this just be added into token.inc?

nod_’s picture

Issue tags: +JavaScript

tagging, I know this one will be involved.

DamienMcKenna’s picture

I thought core already had a browser built-in?

jhodgdon’s picture

RE #40 - Where? There are only a few token functions in token.inc to do replacement, and no UI is provided. That's why there is still the contrib Token module.

jhodgdon’s picture

I have just added an issue summary, since there wasn't one and I believe the current idea would be to adopt the Token contrib module's token browser (which was discussed above and then adopted into the contrib module as noted in #36 above). So, anyone want to make a patch that takes the contrib Token module's UI for tokens and puts it into core?

Gaelan’s picture

FileSize
30.94 KB

Another proposal
Mockup
Explanation:

  • Text box with an arrow next to it
  • When arrow is clicked, display a rules data selector-like dropdown that results in a graphical token.
DamienMcKenna’s picture

@Gaelan: Do we have time to build something like that? How about we start with patching it with Dave Reid's popup token browser from the Token module?

Gaelan’s picture

@DaimenMckenna: I guess we might not. D9 maybe?

nod_’s picture

It's too bad if D8 doesn't have a token browser because we don't have time to work on your proposal when there is an already working implementation somewhere else (that has it's own set of issues but working nevertheless).

I don't have time to work on it but would be happy to review a patch for this.

jhodgdon’s picture

Assigned: Unassigned » jhodgdon

I am working on this.

jhodgdon’s picture

Assigned: jhodgdon » Unassigned
Status: Active » Needs review
FileSize
127 bytes
118 bytes
33.78 KB
PASSED: [[SimpleTest]]: [MySQL] 48,883 pass(es). View

Here is a preliminary, completely untested patch. What I did is to take the bits of the Token contrib module that are related to the token browser UI, and ported them (hopefully) (somewhat) to Drupal 8.

A couple of notes:
- The theme functions need to be converted to Twig presumably.
- There was some caching stuff that I ripped out because caching is different in D8. There are @todo notes in there to indicate where this needs to be put back in.
- I am not sure how to test it. The existing Token contrib module does a bunch of form alters to use tokens... where do we currently want to use it in core?
- There are two PNG files in the patch. I'm not sure that will work, so I also attached them separately.

This is probably about as far as I can take it at the moment... setting to "needs review" so at least the patch will not drop totally into obscurity. If someone has an idea for where this should be used in Core, I could add it there and see if it actually works, and perhaps even add a test... at least for the non-dialog method of displaying tokens. I don't think we can easily test a pop-up dialog using our testing framework.

jhodgdon’s picture

FileSize
36.37 KB
PASSED: [[SimpleTest]]: [MySQL] 48,835 pass(es). View

Dave Reid in IRC suggested adding this to the user email settings form in Core. So I did and it works pretty well! Here's a new patch (was missing a function from the Token project and now it has this added to user.admin.inc).

Things to fix:
- Needs caching as noted in @todo comments.
- Do we need token validation or is that a different issue?
- No tests.
- I noticed that the first time I clicked a token to insert it into a field, it went where I expected (where my cursor was). But then if I clicked another token, it went to the end of the textfield, not where my cursor had been. I don't know how to fix this. daveried wasn't aware of this being an issue in the d7 Token project but it's exactly the same JS code so I expect it is.
- I wasn't sure whether this should really be a separate module or part of system.module... if separate (as in this patch), maybe it should be enabled by default in standard.profile?

Anyway, this is probably ready for a review. nod_ might work on the JS bug?

nod_’s picture

Couple of questions

  • Shouldn't the module be called Token UI?
  • Do we need the insert into textfield functionality? it's kinda weird to have 1 table that messes with several textfields. not having it would take away some tricky code. That'll probably be a pain for a11y as well.

The real question around the JS is do we want the tableTree plugin in core? It has perf issues for big tables (on desktop!) so we can't use it as-is, it'll be terrible on mobile.

I'd argue that we'd want a proper list instead of a table, that would make the JS pretty trivial and that'd scale much better. Also this stuff needs to be able to work on mobile, the current table will likely be problematic.

All in all, it's possible to get the JS in shape and in time, need some UX love and fixing the perf issue somehow.

jhodgdon’s picture

RE #50 -

- We could call the module Token UI, but we have other modules that provide UIs that are not called "something UI", and we currently don't have a Token module, so I decided to go for the "one word if possible" philosophy of core module naming. But I don't care. The other question is, should it just be part of system.module or separate? I don't have a strong opinion.

- I personally think the "insert into field" functionality is a useful UI (minus the current bug that it loses your cursor position)... I have no idea about accessibility though, that's a good point. But having to copy/paste the tokens (or try to type them from a list) does not seem like a friendly UI to me.

- Without the expanding/contract/tree functionality, the full list of available tokens would be LARGE and take up a huge amount of screen real estate. So I think we do need the expand/contract capability.... Is there another UI widget we can use that will accomplish that? I don't know that the table functionality is necessary, so a different tree widget that uses text items instead of a table would likely be workable.

nod_’s picture

The current name supposes you need to enable the token module to enable the token functionality. I don't care either way, just pointing out a possible misunderstanding.

Yep, copy paste kinda sucks for that, agreed. I don't see an issue adding the clicky thing after that got in initially and we figured out the a11y implications and probably other things we haven't thought about yet.

Oh yeah totally I do want collapsible things. It is a must have here. I'm questionning the library and markup we're using since right now it's too slow. didn't feel like a strong use case for a table either, which is why a simple nested list might be workable. Pinged Bojhan about it, hopefully he'll agree :)

Bojhan’s picture

I do not see 3 days sufficient to fix or even attempt to fix the UX, this has gotten designs before those are implemented in the current token UI - and I consider that good enough to iterate upon (adding search, improving the "token" in a field alá facebook, fixing the descriptions). I am more concerned that we might end up exposing this everywhere, something I'd love to avoid.

I do not care if the markup is a nested tree, the only semantic reason I could see for a table is that we have data associated to rows.

jhodgdon’s picture

Another thing to consider with this patch: I took all of the UI components from the contrib Token module... I am not sure we actually need the hook_menu piece, which provides a pop-up window (I think we just want the expandable list/table piece, at least for now?) and the autocomplete (not sure exactly what that is being used for in contrib Token, if anything?)... so those (and accompanying functions) can probably come out of the patch, which will further simplify it.

jhodgdon’s picture

Bojhan/#53 - what do you want to avoid exposing everywhere? The tree table or the token browser?

DamienMcKenna’s picture

FYI the popup dialog is a replacement for the normal token browser that can *greatly* improve performance in many cases as the large tree is neither a) compiled, b) displayed.

Bojhan’s picture

@jhodgdon what is the difference between the two, I guess I am just afraid to find a fieldset below every field :) that could potentially have tokens. However I am sure that worry can be mitigated by a nice toggle or something.

DamienMcKenna’s picture

@bojhan: Install the latest version of Token and Metatag on a D7 site to see the popup interface in action.

nod_’s picture

Turns out we got time now. Better do it right from the start.

The more I see it, the more we need to have tokens in a modal all the time. A link below whatever textarea, input field. Like it's done for metatags in D7. But not the "end of page table" there is on the user page after this patch.

It'd solves the context issue of the "insert into textarea" thing and it'd makes the performance issue less bad.

@Bojhan now that we don't have to rush it as much, can we try out a few things (on g.d.o or something, not this issue) ?

yoroy’s picture

Yeah, this needs some design explorations first. Anyone up to laying out the problem space in a wiki post on gdo/usability? There's enough source materials in the first part of this thread to make a nice briefing of sorts: the problem, the challenges, the constraints and a couple of use cases should trigger some interesting sketches.

yoroy’s picture

Well, for as much as this can be split up in a plumbing part and a UI part it would be good to push forward getting the plumbing in place. We should not 'reset' the effort already put in here. As with the modal/dialog patch, it got committed without a first implementation. Maybe the same can be done here?

DamienMcKenna’s picture

@yoroy: We have time to at least decide the best approach to including this:

  1. Where should it be added - as its own module or to system.module?
  2. Is there an existing library or code pattern that could be used to simplify the code?

I'll be happy to help on the patch, but I'd like consensus on #1 and feedback on #2 before poking it with TextMate.

yoroy’s picture

Good, thank you. Both of those are code/architecture and should be hashed out in here then. Please continue :-)

jhodgdon’s picture

RE #62 item #1:

I think this should be its own module. In many/most cases, using tokens is a site set-up task, and like the Views UI, turning off the token browser module should be an option for an optimized production site (once you've set up the token-related stuff, you should be able to turn off the browser). Also, some people may simply not need the browser. However, I think we should turn on the module in the Standard install profile.

RE: Design...

- Are modal pop-ups accessible? If so (or if we can make them accessible), I think having a modal pop-up is a fine idea, and it would be easy to change the User Email setup to use them (provide a link). The code is already there... I agree that having a large area on every token-using form for the browser is ... clumsy at best. :) If people are interested in seeing the modal, I can make a new patch that uses the modal for the user email fields, and we can try it out. I can also provide screen shots. Let me know if this would be useful.

- I'm also wondering about the modal dialog... if I open the modal, insert a token or two, and then close it, and then I need some more tokens for another text field, could it remember the status (which items were open/closed)? That would be killer!

- Regarding what format is used for the actual browser section (whether modal or on the main page)... The tree table is nice, because it has columns (token/description), but if we had a flat list, I think that would also be fine (after all there are really only two columns that are relevant). So if a flat list tree browser is more efficient and we already have a JQuery library that does it (which??), re-jiggering for that would be fine. If someone points me towards which library, I'd be happy to work up that patch too and we can compare.

nod_’s picture

Design:

1. Modal accessible, yes, in time. It will be made thanks to jquery ui work (they changed their roadmap to fix accessibility things for us).
2. Technically pretty easy. It's a UX thing though and the interaction need to be thought through.
3. I don't know of a library, Collapsible nested lists are pretty trivial to implement, maybe based on the collapsible.js thing since the accessibility has already been figured out for it.

As for the columns if the visual thing is an issue, we can make a lists look like a table, that was already done on the responsive table issue as a proof of concept.

falcon03’s picture

@nod_: could you better explain what you mean for "we can make a list look like a table"?

nod_’s picture

By using the CSS display:table and associated display values (like table-cell, table-row, etc.). As far as I know it's only a visual thing, I'm actually not sure how screen readers handle that.

Unrelated: @falcon03 I'm really happy you're around making sure our stuff is accessible. It's awesome work, thanks! if something needs fixing feel free to ping me, I'll be happy to help out.

jhodgdon’s picture

nod_, Bojhan, jenlampton, and I have discussed the UI for this a bit in IRC and an etherpad. I'll add this to the issue summary too, but here are some notes:

Design considerations

  • Efficient (code/load time - current treeTable implementation breaks with lots of tokens)
  • Accessible
  • Usable
    • Efficient to find the token you are looking for ("scannable")
    • Search capability - which may or may not be different from autocomplete
    • Hierarchical structure of tokens reflected in hierarchical-looking UI (intuitive)
    • Screen real estate (keeping mobile in mind too!)
    • Maybe different modes depending on the scope of the token list?
      - Flat list if there are only a few
      - Tree list and/or hierarchical autocomplete and/or search if the list is longer
      - Not loading the entire list initially if it's really long (page load time)
  • Reusable UI pattern that can be used in Rules, Panels, Blocks, Layouts, Views, Etc. for both tokens and "token-like things".
  • Naming conventions / standards for "token-like things"

Other notes

- TableTree/current implementation is apparently not accessible -- see #922022: The token tree UI is not accessible to keyboard only users on the Token project 7.x
- Token project was exploring other UI possibilities but apparently didn't get too far -- see #1229568: Token UI 2.0

Plan

- Spin off auto-complete and search capaiblity to a separate issue.
- Spin off adaptable modes depending on size of token list to a separate issue
- Token naming should also be separate
- Make this issue only about building a tree-based browser for long lists of tokens
- Use an expandable list instead of the tableTree of the current patch. Core doesn't apparently have a current UI widget we can use for that. nod_ may write one or we could use an existing JQuery component?

jhodgdon’s picture

Issue summary: View changes

Add an issue summary with current proposed resolution, which is to adopt the contrib Token module's UI.

jhodgdon’s picture

I did a search for jQuery tree libraries. It seems that the main actively-developed and available one for now is:
http://www.jstree.com/
(MIT/GPL licensed; accessibility unknown).

The jQuery UI people are also working on a tree plugin but they don't have it done yet apparently:
http://wiki.jqueryui.com/w/page/12138128/Tree

And I also found this one, which is basically an Aria-compatible enhancement of a UL list (sounds promising, and MIT licensed so that is OK):
download: https://github.com/filamentgroup/jQuery-Tree-Control/blob/master/js/jQue...
demo: http://filamentgroup.com/examples/jquery-tree-control/
info: http://filamentgroup.com/lab/jquery_aria_tree_control_from_book_designin...

nod_ also volunteered to write one but if we can use one someone else developed, that would probably be better?

DamienMcKenna’s picture

@jhodgdon: Good research! One thing to point out, though, shouldn't it be an expandable tree-table hybrid because it needs to show columns of data too (description, etc)? There's already work going on in a somewhat-related VDC issue: #1834576: Improve details on the Views listing page.

jhodgdon’s picture

RE #70 @DamienMcKenna - thanks for the link to that other issue. The TreeTable plugin that Token contrib is currently using is not workable (inefficient and not accessible) so we are looking into other possibilities, and right now probably it will be a list. Not sure.

jhodgdon’s picture

Assigned: Unassigned » jhodgdon

By the way I am working on the next iteration...

Bojhan’s picture

Title: Add a token browser UI for core » Add a UI for browsing tokens
jhodgdon’s picture

FileSize
31.99 KB
FAILED: [[SimpleTest]]: [MySQL] 49,004 pass(es), 1 fail(s), and 0 exception(s). View

Here's a proof of concept patch, minus JavaScript... I need some help with that, not familiar with how Drupal does attach stuff?!? Anyway this patch will put up a nested UL list of tokens that you can browse. The links are not working either. I have to run out for a while but I thought I'd put it up in case someone can help with the JS.

DamienMcKenna’s picture

@jhodgdon: A list could work if there were appropriate tooltips to provide the detail listed in the current multi-column display. That said, we could conceivably handle it in two steps: 1. get a basic tree list in place, 2. improve the UI to show the other details.

Status: Needs review » Needs work

The last submitted patch, token-browser-proof-concept.patch, failed testing.

jhodgdon’s picture

Status: Needs work » Needs review
FileSize
35.89 KB
PASSED: [[SimpleTest]]: [MySQL] 49,020 pass(es). View

RE #75 - missing information: The previous patch is only missing the token's name. I've added that now... could be formatted better but at least it's all there.

The other thing that was not working in the patch on #74 was the JS, which I have working now.

Here's another patch (proof of concept only!)... it does work this time!

jhodgdon’s picture

One note on the patch in #77 - the Tree JS thing is not very smart -- it couldn't handle having a DIV inside each level of the item lists. So I (temporarily) made a copy of theme_item_list that omits the divs except on the outer list (so the inner lists are just nested UL LI UL LI UL LI etc., not UL LI DIV UL LI DIV UL etc.).

That should be fixed in the JS, which shouldn't be very hard actually.

jhodgdon’s picture

FileSize
129.48 KB

I forgot, here's a screen shot of the pop-up dialog for the token browser.

DamienMcKenna’s picture

@jhodgdon: Nice work :) I'll give it a review later.

DamienMcKenna’s picture

I took a look at it (via the Menu -> Configuration -> Account Settings -> email message fields) and thought it was really quite nice.

My only concern right now is that with the default theme it takes up a lot of screen space so you don't have a lot of space left on the window to see around it. That said, I think it's definitely a step in the right direction.

falcon03’s picture

@jhodgdon: do you think the patch is ready for an accessibility review?

@nod_: oh, I didn't know this css capability! I'll look for a demonstration of this feature in action and I'll test it with my screen readers, although I don't think it is "parsed" as a table for accessibility...

Unrelated: thank you very much! Developers' availability is very important for me, since actually I don't have the needed skills to roll my own patches (especially for JavaScript, which I don't know at all ;))! Again, thank you very much!

jhodgdon’s picture

RE #82 - well, the patch is more of a "proof of concept" rather than a full patch, but ... yes probably an accessibility review would be useful since we attempted to use a tree plugin that said it was full ARIA/accessible. I am unsure about the dialog pop-up too -- if that is not accessible, we should start thinking about how to make it so and/or an alternative.

Note that the current patch lacks the capability of inserting the token into the field. It just lists the available tokens in a pop-up expandable dialog.

And it has only been added to the email section (vertical tabs) of admin/config/people. To test:
- Install the patch.
- Enable the new Token core module
- Navigate to admin/config/people
- Click the link where it says "Browse tokens that can be used in e-mail messages".

dodorama’s picture

I wonder if we could use a series of nested HTML5 Detail and Summary elements like we did to replace fieldsets.
#1168246: Freedom For Fieldsets! Long Live The DETAILS.
An example of nested elements: http://jsbin.com/egefop/14#html,live

jhodgdon’s picture

Hm. What is that "example of nested elements" link supposed to do? In my browser (Firefox 13 on Linux), it's just a paragraph followed by a single bullet list. Clicking the links does nothing.

jhodgdon’s picture

FileSize
24.26 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch token-browser-proof-concept-v3.patch. Unable to apply patch. See the log in the details link for more information. View
67.6 KB

RE #84 - I made a patch for collapsible fieldsets, but my usual browser (Firefox 13 on Ubuntu, which is a bit out of date) doesn't support them, sigh. Guess it's time to update Ubuntu, which I've been putting off. [Aren't the new display elements supposed to give you something usable in all browsers though if yours doesn't support HTML5??!? Maybe that hasn't been done yet...]

Anyway, I also have Chrome available, which does support them, and they do seem to be a good choice for this. The patch is somewhat nicer (lots smaller!) and the UI seems OK to me. Here's a screen shot and a patch in case anyone else wants to offer an opinion. The token list is a bit bulkier than with the UL lists, but to me the distinction between the expandable items and token items is clearer. And there's no additional JS to load, since these display elements are in core.

jhodgdon’s picture

So... I would be inclined to go for #86. Any other opinions? The patch needs a bit of polish but it's not horrible as it is.

Bojhan’s picture

I am a little bit confused, why we are changing UI's. I thought the idea was only to change the underlaying code.

jhodgdon’s picture

nod_ says that the tree table UI is not workable due to performance reasons, so we are exploring other options.

Also regarding the UI problem I'm having, I filed #1858600: "Filter log messages" details on Recent log messages page applies bogus floats if anyone is interested.

Status: Needs review » Needs work

The last submitted patch, token-browser-proof-concept-v3.patch, failed testing.

jhodgdon’s picture

jhodgdon’s picture

Things that will need to be fixed in this patch:
- Doing drupal_render() in the theme function is not going to work because it doesn't trigger the proper JS and CSS being added to the page.
- The theme functions need to be Twig templates.
- See #1858600-5: "Filter log messages" details on Recent log messages page applies bogus floats - the JS needed in some browsers for the details elements is being properly triggered in the dialog (whether it's a pop-up or in its own page).
- There are some additional @todo notes in the code about caching.

Fabianx’s picture

Great progress here!

DamienMcKenna’s picture

jhodgdon’s picture

Assigned: jhodgdon » Unassigned

Realistically I am not going to have time to work on this again until January at the earliest, sorry! Is there someone else who can pick it up from here and take it to completion please?

Bojhan’s picture

Issue tags: -Needs usability review
rickvug’s picture

Issue tags: -JavaScript, -Usability

Status: Needs review » Needs work
Issue tags: +JavaScript, +Usability

The last submitted patch, token-browser-proof-concept-v3.patch, failed testing.

jhodgdon’s picture

Is someone going to take this over? Is it too late to get this into core anyway?

Dave Reid’s picture

FileSize
24.2 KB
22.87 KB
20.46 KB

Doing a little bit more research on 'drill-down' UIs and came across jQuery Mobile which supports the iOS 'Listview' display, which I think actually may fit very well for our purpose. A sample can be viewed at http://jsfiddle.net/YmcM4/3/embedded/result/ (you have to click the 'Run' button on the top left-ish to see it).

It's kind of nice since we get filtering ability, a very structured way of drilling into more specific tokens (and seeing if the current token offers sub-tokens). It could play well with AJAX since clicking on a list item could load the new "page" via AJAX and shove the generated output into the same request. It works well in a modal. I dunno. I might play with this a little bit more.

The potential problems I see with this so far is that it requires the entire jQuery Mobile library, and for desktop browsers, the UI doesn't really work in a non-modal context.

Available types (first page):

Current date tokens (incomplete):

Node tokens (incomplete):

andypost’s picture

Looks awesome! If jq mobile the only thing that stops - we need it in core

Dave Reid’s picture

Version: 8.x-dev » 9.x-dev

We don't have a good solution, so we've missed our window for D8 at this point. Let's continue to let contrib find the right UI and get it ready to merge into Drupal 9.

DamienMcKenna’s picture

@davereid: Maybe I'm the only one, but that UI doesn't look good to me :p

Also, #sadpanda :(

Dave Reid’s picture

I definitely need to reiterate that the UI in #100 makes sense only used in a modal, so imagine it in a modal only. :) And it also could play well with a theoretical autocomplete functionality that can be used in text fields since the same AJAX that is used to build the 'sub pages' dynamically could also be used to respond with JSON for sub-tokens.

DamienMcKenna’s picture

Given so much has changed in core's JS, are there any similar nested-tree UI widgets elsewhere in core? Maybe we should start there - find a UI widget that fits this use case and others, get it added to core and then add this use case on top of it?

Dave Reid’s picture

There are non in core that I know of. We would still need to find an external library and add it to core.

DamienMcKenna’s picture

I can discuss it with some of the Spark crew this weekend at the SprintWeekend in Boston, see if they have any ideas.

cweagans’s picture

FWIW: I've used http://www.jstree.com/ before and it was pretty decent.

jhodgdon’s picture

RE #108 - does JSTree satisfy the design criteria in the issue summary?

DamienMcKenna’s picture

The website has an unfinished jsTree v1.0 but on github there's a v2 in progress with lots of commits recently.

DamienMcKenna’s picture

Issue summary: View changes

update with current plan

xjm’s picture

Issue summary: View changes
Xano’s picture

Version: 9.x-dev » 8.x-dev
Assigned: Unassigned » Xano
Status: Needs work » Needs review
FileSize
17.41 KB
PASSED: [[SimpleTest]]: [MySQL] 59,319 pass(es). View

As there is consensus that we can keep adding features to Drupal 8 after the initial release, as long as we don't break backwards compatibility, I am re-opening this issue.

The attached patch takes a slightly different approach than the previous one. Instead of porting the Token module's behavior, I decided to lay properly tested foundations first. It is relatively easy to change how the overview is rendered later on (like adding JavaScript), as opposed to changing the reusability after we first wrote the code to be usable for one specific case only.
Next to that, the biggest difference is that since the previous patch was made, the menu routing system has been removed and the token API is now a service. This has its consequences for the code.

This patch is still a work in progress. The token overview page at /token/overview works and most new methods are tested, but some aren't yet. I am not posting an interdiff, as it would be too big to be useful.

Bojhan’s picture

I think it would be nice if you adress how you wish to continue on the steps that @jhodgdon already made in https://drupal.org/comment/6793062#comment-6793062.

I fear that your focus on a nice engineering effort, without vision on how to tackle the UI part will eventually catch up on us - when real users use it. The lack of a good UI was the primary reason why it didn't go into D7. If you wish to get it in, you should figure out how to make it part of tackling this problem. This is not a followup task, this issue is about introducing a UI - therefore it should be good.

tstoeckler’s picture

  1. +++ b/core/lib/Drupal/Core/Utility/Token.php
    @@ -67,6 +67,16 @@ class Token {
    +   *   An array of token types, or NULL when the types are not set.
    ...
    +   * @var array|null
    

    I think the NULL part can be dropped, both from the @var and from the description. Unless they have a default, variables can simply be not set, and in this case that is what happens. The variable is never explicitly set to NULL.

  2. +++ b/core/lib/Drupal/Core/Utility/Token.php
    @@ -305,5 +315,44 @@ public function setInfo(array $tokens) {
    +    if (is_null($this->globalTypes)) {
    

    In accordance with the above, an isset() would be fine here, I think.

  3. +++ b/core/lib/Drupal/Core/Utility/Token.php
    @@ -305,5 +315,44 @@ public function setInfo(array $tokens) {
    +    if (array_key_exists('type', $token_info['types'][$type])) {
    

    I think an isset() would be more standard here. As far as I'm aware having the key set to NULL is not a valid option here.

Didn't get further than that with a detailed review. Looks good though, from a cursory look. I agree with the assessment of putting this into D8. Even if we don't expose it in the core UI, having something in core that contrib can build on that is usable, performant and accessible would be very prudent, IMO.

axel.rutz’s picture

I'm SO SORRY for being late to the party, but here's one addition:

There is one UI option that i could not find in the above thread but should be considered: The rules data selector broser.

IMHO it has some advantages over the other options:
* there is working code to steal
* usability is proven

It might not be the best thinkable but it's there already in terms of code and UX.

I'm working heavily with the token browser and the rules data selector browser.
And everytime i use the token browser i wish i could have the data selector browser.

Just my 5 cents...

DamienMcKenna’s picture

@alex.rutz: I thought that the Rules team were working on a complete overhaul of their UI? And I wouldn't say the "usability is proven" just because Rules has used that UI for a few years.

klonos’s picture

...and we never "steal" - we "borrow" :P

Bojhan’s picture

Yhea, I think you are assuming some usability there - we actually did do Rules testing a while ago for Drupal Commerce and the Data selector part is where most of the problems occurred.

Xano’s picture

I fear that your focus on a nice engineering effort, without vision on how to tackle the UI part will eventually catch up on us - when real users use it.

The main problem with previous efforts is that the code was procedural and did not come with unit tests, and also lacked in reusability; there was no possibility to just render the overview regardless of the context.

I understand that UX is extremely important, but the 'engineers' are the ones who will have to maintain this code and ensure its quality, and who will need to implement it in different contexts. That is why I added a very simple token overview to demonstrate that the code works.

However, any UX and performance advice is not only appreciated, but also actively requested. So if you are experienced in either of these two fields, please share your knowledge here, so we can incorporate it in the patch.

With regards to @jhodgdon in #68, it seems the suggested usability and performance improvements are targeted at two specific parts of the UI:

  1. The token overview: the list of tokens that is used as a reference when you don't know what is available, or what exactly you need.
  2. Form input helpers, such as an autocomplete feature for text input fields that is useful when authors (roughly) know which token they want to use.

If you agree on this, then I propose we make sure these two UI improvements are dealt with in different issues.

Bojhan’s picture

I understand that we wish to have a solid architectural basis, obviously I always support that kind of thinking. I think that jhodgdon outlined it very nicely:

1. Improve the interaction for getting the token browser (e.g. modal, or a collapsed display).
2. Show a teaser list, but not only when the number of token > N.
3. Improve the default token naming (could be separate issue)
4. Improve and build upon the tree based browser currently in contrib (its interaction is rather poor).
5. Allow for auto-complete and/or some kind of search capability.

I'd consider 1,4,5 necessary to get this in and 2,3 improvements that could greatly enhance the usability. I can deal with UI improvements, being in a separate issue - I cannot deal with another half-assed UI in core, that no one cares enough about to fix.

Xano’s picture

Thanks for your feedback! There's one last thing I'd like to address again at this stage: I believe the autocomplete and the token browser/overview are two different things, because autocomplete applies to input elements, whereas a link to open the token browser does not have to be attached to a particular input element per se. What are your thoughts on this?

Bojhan’s picture

Well, ideally you want to be able to both autocomplete in the token field and search the token overview. They are indeed, two separate UI's in rules its current interface they are presented side by side - I expect this not to be the pattern we pursue for core. Instead the browser will be launched either by modal or fieldset.

Xano’s picture

I discussed the inclusion of these changes in Drupal 8.1 or later with @catch and he said "API additions and new features are allowed, FAPI array changes are under discussion".
This means that as long as we develop a solution that does not break APIs and does not add any magic to existing token API implementations (including forms elements that accept tokens), it will be considered.

Xano’s picture

Well, ideally you want to be able to both autocomplete in the token field and search the token overview. They are indeed, two separate UI's in rules its current interface they are presented side by side - I expect this not to be the pattern we pursue for core. Instead the browser will be launched either by modal or fieldset.

I agree that the ideal situation is to have both. What I am trying to establish is whether both additions are technically related or not, so we can split them up into multiple issues and work on smaller patches that are easier to review.

Xano’s picture

On IRC Bojhan agreed that the autocomplete/suggestions and the browser/overview are two separate solutions. I created #2168793: Add token autocomplete support to form elements to deal with the autocomplete, so we can continue working on the browser here.

tkoleary’s picture

Please follow progress on: #2346973: Replace jQuery UI autocomplete with Select2

Changing to Select2 may render this issue obsolete. Select2 has better ways of handling tokens.

Dave Reid’s picture

@tkoleary: I think that statement ignores the complexity we've had to face from exposing a UI in contrib, with all the nested information.

tkoleary’s picture

The nested information does not need to be exposed, and in fact is a barrier to usability because of information overload.

Here's how it might look with Select2 in site email config UI: http://invis.io/2T1HGEWZJ

The select2 features this might utilize are:

  • Multi-value select boxes (for the pills inside the text area)
  • Placeholders
  • Loading Array Data (possible substitute for using ajax)
  • Programmatic Access or Reacting to external value changes (for resetting the pill when a suffix is added)
  • Drag drop sorting (a nice-to-have if we want users to be able to move the tokens around the text area)
  • Responsive Design - Percent Width (for mobile)

Here's the documentation for reference:
http://ivaynberg.github.io/select2/index.html

tkoleary’s picture

The nested information does not need to be exposed, and in fact is a barrier to usability because of information overload.

Here's a linear mockup of how it might look with Select2 in site email config UI:

http://invis.io/2T1HGEWZJ (hold down the shift key to expose what's clickable in the mockup)

The "add suffix" placeholder should only appear when there are children, and I'm not sure that's the right nomenclature.

I hallway tested this on a few members of the Acquia support staff who use this UI commonly and all found it a big improvement. My hypothesis is that if a user comprehends the use of tokens they are likely to have a good idea which one they are looking for to start typing and the category labels help refine that.

Select2 features this could potentially utilize are:

  • Multi-value select boxes (for the pills inside the text area)
  • Placeholders
  • Loading Array Data (possible substitute for using ajax)
  • Programmatic Access or Reacting to external value changes (for resetting the pill when a suffix is added)
  • Drag drop sorting (a nice-to-have if we want users to be able to move the tokens around the text area)
  • Responsive Design - Percent Width (for mobile)

Here's the documentation for reference:
http://ivaynberg.github.io/select2/index.html

DamienMcKenna’s picture

@tkoleary: That would work for two or maybe three levels of depth, but what about accessing data from entities tied into the current entity via reference fields? It's an interesting idea, but it would simply need a way of digging into the detail further.

tkoleary’s picture

@DamienMcKenna

There seem to be a few different methods in Select2 for loading data. Maybe one of them could be used? Pinging Nod, he made the select2 patch for language selection.

nod_’s picture

I like the idea for tokens in textarea but we can't do that only with select2. The prototype is showing what CKEditor widgets are. We can't implement the design without taking control of everything in the textarea, which means implementing most of what a WSIWYG editor does.

Short of putting ckeditor in those textareas and having token widgets the design is not doable. The input search field looks good though.

tkoleary’s picture

@nod

I figured there might be a problem with getting the tokens to display as pills, but in fact that is really a nice-to-have and the usability improvement is not really diminished if we just have them appear in square brackets as they do now.

Leeteq’s picture

(PS. I have not read the whole thread in detail, just searched the page for "role" and "permission")

Out of curiosity;

Performance-wise, would it be useful to have role-based ability to load whole sub-trees?

All users could be served all tokens down to level 2 or 3 on the initial page load.

Users with role X could then be permitted to load whole sub-trees below that limit with one click (on the desired level/group), using AJAX inline load so that the whole page does not refresh.

Users without that role would need to click on each level and wait for the same AJAX to load one level leeper on each click, yet still without loading the same page.

That would combine elegancy and flexibility with performance concerns.

jhodgdon’s picture

Version: 8.0.x-dev » 8.1.x-dev
Category: Task » Feature request
Status: Needs review » Postponed

"Add a UI" is a feature request... so this really does need to be postponed to 8.1.x at this point, and/or contrib, in spite of the fact that it would be really nice to have in core.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Berdir’s picture

Status: Postponed » Needs work

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

yoroy’s picture

Assigned: Xano » Unassigned
Issue tags: +Needs design