Placeholders not available when using advanced (token) mode

Added the code to have the "Placeholders available" section for the Advanced mode. Needs to be reviewed.

Hi
Tested MR 621 on Drupal 11.x
The Mr is applied successfully...

Testing Steps:

1. Install AI Module
2. Install the AI Automators module with the Token Module
3. Enabled the AI Autometers on the entity type's field
4. Select AI Automatos settings > Automator Input Mode > Advanced Mode (token)
5. Check for the Placeholder

Result:
The Placeholder is available when using advanced (token) mode.

Attaching screenshots for reference

I think it would be better if Marcus could confirm whether the placeholders not implemented for advanced mode are done on purpose or missed.

It wasn't on purpose to leave them out, it never struck me that you could need them. I'll change this to task and set in to 1.2.x and add it to the roadmap for 1.2.0.

Tested the feature. Issues I Observed:

1. The placeholders are available in automator for advanced mode. but what does this placeholders stand for?
   

   {{ context }} - The cleaned text from the base field.
    {{ raw_context }} - The raw text from the base field. Can include HTML
   { max_amount }} - The max amount of entries to set. If unlimited this value will be empty.

   These explanations are valid only when we have selected a base field. but in advanced mode, there is no base field selected.

2. Enabled automator for a field, Selected "Advanced Mode", used prompt "write an article on {{ context }}." and saved the setting. But did not get any response.
3. Enabled automator for a field, Selected "base mode" and selected a field title as "Automator Base Field", then changed the mode to advanced mode and used the same prompt as above "write an article on {{ context }}." and saved the setting. Now I got ai response in the field with an article on title field(selected as base field).

Working on the feedback

I attempted to clean up the direct calling of services in the RuleBase file and quickly realized that it would be a significant task. The RuleBase has been extended by many other classes, each with its own dependency injection for the services included in the RuleBase. I will create an issue for this cleanup since I have completed the current task.

I feel there is a bug in the logic how the form handle the Automator Base Field value. Event if we select Advanced Mode (Token) the value for Automator Base Field get stored in the config. Which result it making the Placeholders context work in some cases. In theory Placeholders context shouldn't be need for the advance mode.

I feel the solution here will be to remove the placeholder for the advance mode and manage the value of Automator Base Field addConfigValues() to only save the value when the mode is Base Mode.

// Only persist base_field when Base mode is selected; otherwise clear.
$aiConfig->set('base_field', ($formState->getValue('automator_mode') === 'base') ? ($formState->getValue('automator_base_field') ?? '') : '');

Advanced Mode is intended to be used only with tokens, that's how I feel.
Changing this needs review, as it requires input from the maintainer of the module regarding the logical implementation standpoint for this functionality.

There are some advantages with the contexts the Automator sends - there are for instance in the taxonomy automator type, ways of getting massages lists of taxonomies etc.

I wonder though, if we are just band-aiding an work around - I think maybe we should postpone this to 2.0 and actually bring the specific context placeholders into tokens and utilize the prompt library we are adding in 1.2.0?

The thing that is possible in base mode, that might not be utilized much, is that it is rendered via Twig. But we should push for this being possible to use in the prompt library instead, so we have the possibility to for instance show one dynamical value conditional on another or loop items.

I'll remove beta blocker for now and I think this is a better candidate for full rework in 2.0?

Since 2.x is still WIP, I think it makes sense to add a fix here. Automators have not changed much in 2.x, so we should be able to make a fix and forward port to 2.x rather easily. This should go primarily in to 1.3 though, I don't think we have to backport it for older versions.

Added MR !1352 targeting 1.3.x. Builds on the UI fix from 621 (showing automator placeholders in token mode) and adds the actual processing.

RuleBase::generate() in token mode now runs renderPrompt first on the original template, and then token replacement.
Also guards generateTokens() against empty base_field, and includes a unit test.

Adding a patch for use as an immutable object with composer patches

It seems a bit weird to run regular token replacements + twig-processing on the same input.

Could we instead rely on token replacements only, since we already do that, and add additional placeholders to the tokens a 'string' $data? It should be possible to pass them via to the token replacement/info API as part of $data:

array $data: (optional) An array of keyed objects. For simple replacement scenarios 'node', 'user', and others are common keys, with an accompanying node or user object being the value. Some token types, like 'site', do not require any explicit information from $data and can be replaced even if it is empty.

Then, I'm not sure about to handle "placeholders" relying on a base-field best. The ones we provide, like context, I'd filter in token mode instead of noting it in the help-text. But that does not solve the problem for possible placeholders provided by sub-classes. Any thoughts on this?

Moving to 1.4.x.

One major problem was found.

Also we need to document this update in documentation.

This code review is a pre-review performed by AI. A secondary human review will be done, when it passes AI review.

Hey @petar_basic

Thank you for your work on this and for moving it forward. Could you please check and let us know if it is ready for review?
Also,
It would also be really helpful if you could cross-check the testing steps as well.

Reworked per Wolfgang's feedback. It now uses the Drupal Token API instead of mixing Twig and token replacement. Automator type plugins provide dynamic tokens as [ai_automator:*] in token mode, so a prompt can combine entity tokens with automator data, e.g.: Based on [node:title], choose from [ai_automator:value_options_nl]

Also addressed Marcus's XSS feedback, added documentation, and included a unit test.

MR !1423 (1.4.x) and !1352 (1.3.x) are ready for review. CI green on both.

Testing steps:
1. Enable ai_automators and token modules
2. Go to Structure > Content types > [type] > Manage fields
3. Edit a field with a compatible automator type (e.g., taxonomy entity reference)
4. Enable AI Automator, switch to "Advanced Mode (Token)"
5. Verify "Automator tokens available" section appears showing [ai_automator:*] tokens (but not context/raw_context)
6. Set a prompt using both entity tokens and automator tokens, save, and trigger the automator
7. Verify the AI receives the prompt with all tokens replaced by actual values (e.g. have AI logging on)