Merge Token and Rep[lacement]Tags module

I'm author/maintainer of the Rep[lacement]Tags module. As mentioned before reptag and token are VERY similar in the API they expose, what means reptag can (almost) do everything token does (and this for quite a long time). I guess modules are using token for the simple reason that is was promoted/developed by a popular and skillfull developer. However its frustrating to maintain a module that on one hand is loved by several people for a long time but on the other hand rivals with one of the most popular modules i.e. token. Thats a hopeless competition for me ;)

Is there any chance that token will use regexps shortly? So that it will support more sophisticated replacements patterns? Any chance to merge the modules or any way for them to supplement each other?

I'm asking because I'd love to transform reptag into a module that only implements what token does not:

• Provide an interface to configure tokens to be used in a filter for nodes and cck fields. (I know in the meantime even cck started to support tokens.)
• Provide a plugin framework so that users can easily expand token's capabilities. It's actually the most popular feature in reptag, that users can simply drop in a .tags module (with a very simple syntax even non-developers can adapt for their purposes).
• Add the ability to manage replacements (table tags in reptag) online from the admin pages (second most popular feature in reptag).

All in all reptag could be the UI and filter module for token. No more, no less.

@greggles: You ask me to do a presentation about reptag in Barcelona and to talk with you and Eaton about merging the modules, but unfortunately I wasn't able to attend DrupalCon. Do you think we can talk about this over the net (per email, issue, irc)?

... though I'll need to learn more about the ".tags" system before I can comment on that

The '.tags system' is simple to understand:
Token includes files containing 'tag => replacement' pairs (i.e. token_node.inc, token_user.inc etc.) in a static fashion. The .tags files in reptag are very similar to these token_* includes, but reptag implements a registry to include these files dynamically. You can simply put a new .tags file in the folder and reptag automatically discovers it and provides you options to configure. Configuration consists of two parts mainly: 1. enable/disable includes (similar to the 'Modules' screen (admin/build/modules) of Drupal core) and 2. it allows you to include the tags on a per-role basis (for reptag users a very appreciated feature, because the build-in (core) filter system does not allow to specify granular permissions for filters). I guess you never tried reptag, otherwise the idea behind this system is very easy to grok.

Also, I believe with some use of hook_token_list and hook_token_values in the reptags module you could already merge them at least 75% of the way so that people could use reptags features from token. Even that would be a good achievement.

As mentioned above I think I could wrap reptag around token quite easily, especially to provide a UI. However I dont think it is possible already for the simple reason, that Token does not support regular expression syntax to do replacements. Thats a big deal as about 40% (guesstimated) of reptag's tags will not be portable. Reptag allows much more dynamic replacement patterns than str_replace() can handle.
The easiest and most flexible solution for this would be a way to 'override' str_replace(), so that reptag could simply drop in its own mechanism to handle regexps itself. We could simply add a property to the $tokens array (returned from hook_token_values), for example $tokens['#process'] = 'mymodule_token_process'; (similar to FAPI), where the 'mymodule...' function would be called (if specified) instead of str_replace(). This would nicely add to the existing token syntax and would allow to process sophisticated replacement patterns.
A mechanism like this should be simple to implement (in 10 lines or so) and it should be lightweight as it does not add much overhead to the token handling. However this small addition would indeed allow me to port 75% of the reptag features to a token-based solution. A really promising perspective IMO.

One thing that cant be easily ported to token is the node filter stuff, that allows users to filter the node content on a per content type (and even per field) basis. However I think thats more a nice addition and we can live without it happily. I still hope that the filter system in core will be revised to support this features (per content type at least) by default.

I hope you (token developers) will think about the idea of merging reptag and token (or more precisely allow me to wrap reptag around token). Maybe you will have time to play with the Drupal 5 release of the Rep[lacement]Tags module for 10 minutes or so as it would surely help you to understand my points. Reptag could be a really nice supplement for token and its users.

To sum up ... I would drop the node filter stuff from reptag and wrap all other features around the token infrastructure. This means token would instantly gain the following features if reptag is installed:

1. Table replacement tags:
   SiteWide- and User-Tags are managed from the reptag administration pages. You can easily add/delete/modify these tags online. All table tags are also language-aware ('Locale' and/or 'Content translation' modules in D6 supported). The new reptag version for D6 will ship with AJAX/AHAH-powered tags administration.
   - SiteWide replacement tags can be used by everyone on your site.
   - Every user with 'manage user reptags' permission can additionally define his/her own user replacement tags. The user-defined tags are available only for content created by the user and they can override site wide tags by default.
2. The .tags system to allow dynamic extension with additional tags from user .tags 'modules' using a simple regexp-based syntax. Inclusion of these tags on a per-role basis. Thats about 80% of reptag's functionality and should be a huge push for more sophisticated tokens.

What do you think?

Edit: I just found the 'Tokenize' module (http://drupal.org/project/tokenize), which implements the content filter stuff for token already (although the code needs a little love IMO).

Just to notice, that doing a rewrite of RepTag for Drupal 7 I'm currently trying to integrate best with token in D7 core. However, as the concept of tokens changed drastically in preparation for core, its not that simple anymore ... I will comment here again once RepTag 7.x is available for testing.