Problem/Motivation

Currently, the assignment of default values is hardcoded for the following operation types:

  • chat
  • chat_with_image_vision
  • chat_with_complex_json
  • text_to_image
  • embeddings
  • text_to_speech
  • speech_to_text
  • moderation

The 1.1.x version of the AI module has some additional operation types:

  • Chat with Structured Response
  • Chat with Tools/Function Calling

These additional ones needs to be handled or would be better if we can have this list of operation type dynamic so that every time there's is modification to these could be handled automatically.

Steps to reproduce

modules/contrib/ai_provider_openai/src/Form/OpenAiConfigForm.php

 $this->aiProviderManager->defaultIfNone('chat', 'openai', 'gpt-4o');
    $this->aiProviderManager->defaultIfNone('chat_with_image_vision', 'openai', 'gpt-4o');
    $this->aiProviderManager->defaultIfNone('chat_with_complex_json', 'openai', 'gpt-4o');
    $this->aiProviderManager->defaultIfNone('text_to_image', 'openai', 'dall-e-3');
    $this->aiProviderManager->defaultIfNone('embeddings', 'openai', 'text-embedding-3-small');
    $this->aiProviderManager->defaultIfNone('text_to_speech', 'openai', 'tts-1-hd');
    $this->aiProviderManager->defaultIfNone('speech_to_text', 'openai', 'whisper-1');
    $this->aiProviderManager->defaultIfNone('moderation', 'openai', 'omni-moderation-latest');

Proposed resolution

TBD

Remaining tasks

TBD

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#5 config-ai-settings.png359.52 KBishani patel

Issue fork ai_provider_openai-3521901

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

prashant.c created an issue. See original summary.

prabha1997’s picture

prabha1997 commented
Assigned: Unassigned » prabha1997

prabha1997 opened merge request !36

prabha1997’s picture

prabha1997 commented
Assigned: prabha1997 » Unassigned
Status: Active » Needs review

I’ve added a dynamic implementation to set the default provider and model ID for each operation type.

ishani patel’s picture

ishani patel commented
StatusFileSize
new config-ai-settings.png359.52 KB

I've checked with 1.1. x of the AI branch, and it is working as expected. Please refer to the attached screenshot.

Thank you!

prashant.c’s picture

prashant.c
he/him
Primary language English
Location Dharamshala
commented
Status: Needs review » Needs work

Left some review comments, moving to NW. Kindly address those and move back to NR.

prabha1997’s picture

prabha1997 commented
Status: Needs work » Needs review
marcus_johansson’s picture

marcus_johansson commented

I don't think they should be dynamically set in the providers with known choices - these are editorial choices of models that fit as the best possible model for tested use cases like Drupal CMS agents, but still cheap. gpt-4 was always better than gpt-4o on text tasks, but we still choose 4o because it was faster and cheaper.

In many providers we do not even set these, because they are not up to the standards for every operation type.

The question is if its time to bump up all the chat_with_complex_json, chat_with_structured_json, chat_with_tools to 4.1 since its cheaper and better (not tested with Drupal CMS yet).

WDYT?

Techincally the code looks good, just wondering if it makes sense since the order coming back could in theory pick 3.5 turbo on json and tool calling, since its possible.

marcus_johansson’s picture

marcus_johansson commented

A way to cleanup code is to add it here and then link to this in the form?

prabha1997’s picture

prabha1997 commented

Thank you for the feedback! As of now, I’m getting the default models from the plugin and have set them dynamically like this. Please review this approach. I understand your point about editorial choices, and regarding bumping models to GPT-4.1, we can wait until the tests with Drupal CMS are completed. Once we have more insights, we can make the necessary changes.

marcus_johansson’s picture

marcus_johansson commented

I did some initial tests with 4.1 vs 4o and they work more or less the same on the small data set I've tested against: https://docs.google.com/spreadsheets/d/1IfWlRhlq2E4Lh5R53W3qicink9eDzi6U.... The failures happening are code or prompt failures, rather than provider issues.

Since 4.1 is cheaper I think at least for chat_with_tools we could set it?

prabha1997’s picture

prabha1997 commented

Hi @marcus_johansson,
I'm currently unable to access the spreadsheet — it looks like I don't have permission. Could you please grant me access?
Thanks!

marcus_johansson’s picture

marcus_johansson commented

Could you make a request for it or write your e-mail (on Slack if you don't want to publish it here)?

prabha1997’s picture

prabha1997 commented

Hi @marcus_johansson,
Thanks for sharing access — I’ve checked the sheet. While I understand the data, I’m not yet at the level where I feel confident analyzing and comparing the cost-effectiveness between 4.1 and 4o in depth.

marcus_johansson’s picture

marcus_johansson commented
Status: Needs review » Fixed

Thank you, getting merged!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.