Problem/Motivation

Config-entity AI Agents (ai_agents) drive tool use via native function calling. They call ChatInput::setChatTools() and read tool callsback from $response->getNormalized()->getTools(). CliProvider currently ignores the provided tools and never returns any tool calls, so the model has no way to invoke tools such as ai_content_review_result. In the AI Agent Explorer, this surfaces as: I'm unable to call ai_content_review_result because that tool is not available in this environment.

As a result, the CLI provider cannot be used with ai_agents or ai_content_review at all, even though the underlying CLI models are fully capable of producing structured output.

Steps to reproduce

  1. Using drupal/ai:^1.4.6, ai_agents, ai_content_review, and ai_provider_cli:^1.0-dev on drupal/core-*:^11.4.
  2. Create and enable an AI Agent that requires a tool (for example, the ai_content_review_result tool).
  3. Select the CLI provider for the agent and run it in the AI Agent Explorer.
  4. Observe that the model never calls the tool and the agent reports that the tool is unavailable.

Proposed resolution

Since CLI tools do not expose a native tool-calling API, emulate function calling at the prompt level within ai_provider_cli:

  • Advertise the AiModelCapability::ChatTools capability.
  • When ChatInput includes tools, serialize the OpenAI-style tool JSON schema into the prompt and instruct the model to respond with a strict envelope: "tool_calls":[{"name":"<tool_name>","arguments":{…}}]}
  • Parse the envelope from CLI stdout and construct ToolsFunctionOutput objects, setting them on the normalized ChatMessage. This should mirror how native providers such as ai_provider_aws_bedrock populate tool outputs.
  • Add kernel test coverage for the tool-call round trip and for the "no tools → output unchanged" case.
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

panchuk created an issue.

  • 44c0f13d committed on 1.0.x
    Merge branch '3616342-add-function-calling-tools' into '1.0.x'
    
    Issue #...

  • ba68a948 committed on 1.0.x
    Issue #3616342: Add function-calling (tools) support.
    
aangel’s picture

@panchuk I merged this after a test with Claude showed it working. Then I thought to check it with a model less capable than Claude with agents and it failed. And that led to this document Claude put together for me as I was exploring how to handle less-capable models:

# Tool-calling reliability
https://git.drupalcode.org/project/ai_provider_cli/-/raw/1.0.x/docs/tool...

And here is the test harness:
https://git.drupalcode.org/project/ai_provider_cli/-/blob/1.0.x/docs/tes...

Rather than reverting the merge, I'll keep it in so that people can play with it. But we will need more work on this to make it work across more models.

I'll keep this issue open, too.

panchuk’s picture

Thanks for review and testing this.

Before installing ai_provider_cli I was playing with ai_provider_lmstudio and noticed that LM Studio not able to execute tools properly. I'm not sure that is model problem, since I tried llama3 70B and qwen2.5 70B Instruct, but rather the LM Studio implementation of the tool calling.

TBF I missed something with setup of LM Studio

aangel’s picture

Status update: tool calling ships in 1.0.0-alpha1, labelled experimental in the README. As of MR !6 it no longer fails silently: when a model replies with something other than the tool-call JSON, or calls a tool it was not offered, the reply is kept as plain text and a warning is logged with the start of the reply. That should make the weaker-model failures in docs/tool-calling-reliability.md visible in the log.

Leaving this open for the reliability work across more models.