Problem/Motivation
I'm coding a new ai ckeditor plugin to allow editors to send a file (pdf or image) to mistral AI to fill the text editor with the file text content using the ocr capabilities of the language model. (https://mistral.ai/fr/news/mistral-ocr)
For this to work in a development environment, I need to be able to send files to the ai service on /files endpoint. (https://docs.mistral.ai/api/#tag/files/operation/files_api_routes_upload...)
The current implementation of the provider do not offer this possibility.
Proposed resolution
Add missing methods in MistralClient.php and MistralProvider.php files.
Remaining tasks
Provide patch or create a merge request
User interface changes
none
API changes
These two methods are available :
$file = \Drupal::entityTypeManager()->getStorage('file')
->load($fid);
$fileInfo = $provider->uploadFile($file, 'ocr');
$fileUrl = $provider->getFileUrl($fileInfo['id']);
File url provided by mistral can be used in a chat message.
$messages = [
[
'role' => 'user',
'content' => [
[
'type' => 'text',
'text' => "Extract the content of the following file. Do not add any comments and do not modify the content.",
],
[
'type' => 'document_url',
'document_url' => $fileUrl,
]
]
]
];
$config = [
"max_tokens" => 2048,
"top_p" => 0.4,
];
$provider->setConfiguration($config);
/** @var \Drupal\ai\OperationType\Chat\ChatMessage $response */
$response = $provider->chat($messages, $model, [
'vdg_ai',
])->getNormalized();
$content = $response->getText();
| Comment | File | Size | Author |
|---|---|---|---|
| mistral-file-upload.patch | 3.11 KB | olivier.br |
Issue fork ai_provider_mistral-3515048
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
olivier.br commentedComment #3
petar_basic commentedThis needs rework due to the change in the PHP library this module uses.
Comment #5
petar_basic commentedComment #6
fagoLooks mostly good, just found one remark. I think this also caused the CI fail. Let's also add one basic integration text for files?
Comment #7
petar_basic commentedCorrected and added a test. Please check
Comment #8
fagoGenerally the MR looks fine and makes sense, but I realize just now that this is implementing a mistral-specific API without having a Drupal-ai generic interface for this capabilities. I'd love to hear Markus' thoughts on adding provider-speciific logic first.
Comment #9
petar_basic commentedOk, as per Markus' answer in slack, I've added support for the generic Drupal AI interface in the chat() method:
- Changed from getImages() to getFiles()
- Added handling for documents alongside images
- Both sent as base64 data URIs
This change just makes the provider work with the standard Drupal AI interface that other providers (like OpenAI) also use, so files can be attached via ChatMessage::setFile()
I've still kept the file management methods (uploadFile, getFileUrl, listFiles, etc.) for any custom/advanced use cases.
Also fixed: Uninitialized $pages property error in responseToArray().
Comment #10
fagolooks good, this is the really valuable feature - support for attaching files in chats!
MR needs a re-roll though.
Comment #11
fagoComment #12
petar_basic commentedUpdated MR & fixed phpstan issues
Comment #14
fagoall good. I had resolve conflicts, merged.