Implement the UI data selector autocomplete

There was too much for me to learn about Drupal 8 and how Rules worked for me to complete much.
My original idea was to (a) learn autocomplete, (b) pass in / retrieve context, (c) use context (with conditions) to generate / filter possible autocomplete matches.
I got stuck in (b) because I could pass in some information but not retrieve the entire context for the rules and conditions. Even ignoring that, I was not able to proceed due to some issues with rules validation.

I've added a link to do a small diff to show what I've learned for anyone else that may wish to continue.

https://github.com/fago/rules/compare/8.x-3.x...KingJoshi:2648326-data-s...

* We need to add a route in rules.routing.yml for rules.autocomplete
* Form element needs two new options "#autocomplete_route_name" and "#autocomplete_route_parameters" to trigger autocomplete
** "#autocomplete_route_name" is the route created in rules.routing.yml
** "#autocomplete_route_parameters" is an array of parameters for the url
* I made it point to a new autocomplete function in src/Controller/RulesController.php
* Function returns return new JsonResponse($matches); where $matches is an array of arrays of the format:
** [ ['value'=>'value1','label'=>'label1'], ['value'=>'value2','label'=>'label2'] ]
* I was not able to retrieve the current form using form_build_id in Drupal8
* there were validation issues with Rules at this time
* The data selector concept seems simple for doing autocomplete, but the actual UI seems confusing for new users. I would have assumed autocomplete for the direct values. For example, if I'm trying to match a user to a role, then autocomplete should be of the roles
* I do pass in the mode (selector/input) in to the autocomplete function. It's not clear to me what information needs to be passed and what can be regenerated (very unfamiliar with Drupal 8 Forms)

I think duplicating that logic would be very bad, as it will become even more complex in future with metadata assertions etc. So checkIntegrity() should re-use the generally available Metadatastate, which has been processed etc.
In Drupal 7, there was something like $expression->getAvailabileVariables() which did all the calculation and returned you the info about available variables based on the metadata execution / processing UNTIL the expression. So it asked the parent expression about state + added in all variables / metadata assertions of sibling expressions being executed earlier.

proccessMetadataState() sounds exactly like that in d8, so I assume the only difference is that it would pass around the metadata state instead of keeping it internally. But we cannot easily reverse-the-flow with asking parents as we did in d7, as we do not have the (cyclic) parent relationship. But I do not really see another way to implement that, so it looks like we have to add that $parentExpression variable and cyclic object references. People are more used to them anyway now as entities have them also :D Or do you have another idea?

thanks a lot for the great progress here :)

Looking at the code, it looks good overally and I'm happy to see decent test coverage.

Here are some comments:

class RulesComponent
  public function autocomplete($partial_selector, ExpressionInterface $until = NULL) {
    // We use the integrity check to populate the execution metadata state with
    // available variables.
    $metadata_state = $this->getMetadataState();
    $this->expression->prepareExecutionMetadataState($metadata_state, $until);

    return $metadata_state->autocomplete($partial_selector);
  }

This seems to repeat API which is on the metadata state anyway. I think, instead we just miss an API function gets the metadata prepared for a certain $until element. Then you can do whatever necessary with that metadata state, e.g. get autocompletion results or other data information as needed for the UI.

ad \Drupal\rules\TypedData\DataFetcher::autocompletePropertyPath:
The separation into first,middle and last part confused me bit. I realized then, that it takes care of find the first variable while other methods like fetchDataByPropertyPath() take the starting variable as first parameter. It should follow that pattern and if necessary separate the part of finding the right variable in another method.

ad

    $middle_path = implode('.', $parts);

    if ($middle_path === '') {
      $variable_definition = $data_definitions[$first_part];
    }
    else {
      try {
        $variable_definition = $this->fetchDefinitionByPropertyPath($data_definitions[$first_part], $middle_path);
      }

this should use fetchDefinitionBySubPaths() and save the imploding.

  protected function getAutocompleteSuggestion(DataDefinitionInterface $data_definition, $variable_name) {

If it can be a property path, the variable name should be $property_path not $variable_name.

Any progress here? Just checked Rules again today and no autocomplete suggestion. This appears to be the make or break feature to make Rules usable via the UI.
Maybe the progress is happening somewhere else? Happy to be pointed there.
----- Edit -----
Autocomplete Suggestions work, just somehow did not see it / get it to trigger.

@eigentor: I'm not sure what you are expecting to see, but the autocomplete suggestion code was committed more than two years ago, as per the above, and has working in Rules since then ... Note this is for the data selection mode only, not for direct input ...

This should probably be moved to Typed Data API Enhancements module now? The DataFetcher was moved, but the autocomplete test is still in Rules ... I think the JavaScript and the tests should probably be moved too - Rules is just a consumer of that API.

Sorry for not reporting back. I meanwhile discovered the autocomplete is working fine. Somehow did not get to trigger it before. So forget my comment.