Problem/Motivation

The Vertex provider is not properly interfacing with Google's Vertex AI API schema. When setting message content, you may only specify user or model https://cloud.google.com/vertex-ai/docs/reference/rest/v1/Content.

Incorporating a tools response, $message->getRole() will be tool. This causes a 400 response.

https://git.drupalcode.org/project/ai_provider_google_vertex/-/blob/1.1....

Steps to reproduce

Create an agent that uses a tool, have a conversation that causes a tool to be used, get an exception.

Proposed resolution

Make the code more robust and strictly adhere to the Vertex AI API spec.

Remaining tasks

  • Update the provider chat implementation to properly identify tool usage and format the payload accordingly.
CommentFileSizeAuthor
#3 3530565-tool-calling-fails.patch2.78 KBlpeabody

Issue fork ai_provider_google_vertex-3530565

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

lpeabody created an issue. See original summary.

lpeabody’s picture

lpeabody commented
Issue summary: View changes
lpeabody’s picture

lpeabody commented
Status: Active » Needs review
StatusFileSize
new 3530565-tool-calling-fails.patch2.78 KB

yce made their first commit to this issue’s fork.

yce opened merge request !10

mrdalesmith made their first commit to this issue’s fork.

mrdalesmith’s picture

mrdalesmith commented

I've tried to fix the failing tests on this, but cspell is an abomination and will not ignore the word "blacklist" which isn't in the project but the word "git_blacklist" does aoppear in the grumphp.yml file: presumably there is someway of getting cspell to ignore this file?

There was an additional issue in the Provider codebase that caused an error when I tried to use tool calling with the AI Provider: all tool parameters were being added as required fields, which meant any optional fields caused an error if they were missing from a valid AI response. Fixing that, and I have managed to get the AI Assistant to create a new content typed called "Tester" from the prompt "Create a content type called Tester" without erroring.

yce’s picture

yce commented

I've found a way to ignore the forbidden word, pushed to the MR.

lpeabody’s picture

lpeabody commented
Status: Needs review » Reviewed & tested by the community

Confirmed this is fixed by replacing my original patch with the MR diff in my composer.json.

  • yce committed 20af5873 on 1.1.x 
    Issue #3530565 by lpeabody: Tool calling does not work
yce’s picture

yce commented
Status: Reviewed & tested by the community » Fixed

Now that this issue is closed, please review the contribution record.

As a contributor, attribute any organization helped you, or if you volunteered your own time.

Maintainers, please credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.