Problem/Motivation

I'm coding a new ai ckeditor plugin to allow editors to send a file (pdf or image) to mistral AI to fill the text editor with the file text content using the ocr capabilities of the language model. (https://mistral.ai/fr/news/mistral-ocr)

For this to work in a development environment, I need to be able to send files to the ai service on /files endpoint. (https://docs.mistral.ai/api/#tag/files/operation/files_api_routes_upload...)

The current implementation of the provider do not offer this possibility.

Proposed resolution

Add missing methods in MistralClient.php and MistralProvider.php files.

Remaining tasks

Provide patch or create a merge request

User interface changes

none

API changes

These two methods are available :

$file = \Drupal::entityTypeManager()->getStorage('file')
      ->load($fid);
$fileInfo = $provider->uploadFile($file, 'ocr');
$fileUrl = $provider->getFileUrl($fileInfo['id']);

File url provided by mistral can be used in a chat message.

$messages = [
      [
        'role' => 'user',
        'content' => [
          [
            'type' => 'text',
            'text' => "Extract the content of the following file. Do not add any comments and do not modify the content.",
          ],
          [
            'type' => 'document_url',
            'document_url' => $fileUrl,
          ]
        ]
      ]
    ];
$config = [
    "max_tokens" => 2048,
    "top_p" => 0.4,
];
$provider->setConfiguration($config);
/** @var \Drupal\ai\OperationType\Chat\ChatMessage $response */
$response = $provider->chat($messages, $model, [
  'vdg_ai',
])->getNormalized();
$content = $response->getText();

CommentFileSizeAuthor
mistral-file-upload.patch3.11 KBolivier.br
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

olivier.br created an issue. See original summary.

olivier.br’s picture

Issue summary: View changes
petar_basic’s picture

Assigned: Unassigned » petar_basic
Status: Active » Needs work

This needs rework due to the change in the PHP library this module uses.

petar_basic’s picture

Assigned: petar_basic » Unassigned
Status: Needs work » Needs review
fago’s picture

Status: Needs review » Needs work

Looks mostly good, just found one remark. I think this also caused the CI fail. Let's also add one basic integration text for files?

petar_basic’s picture

Status: Needs work » Needs review

Corrected and added a test. Please check

fago’s picture

Generally the MR looks fine and makes sense, but I realize just now that this is implementing a mistral-specific API without having a Drupal-ai generic interface for this capabilities. I'd love to hear Markus' thoughts on adding provider-speciific logic first.

petar_basic’s picture

Ok, as per Markus' answer in slack, I've added support for the generic Drupal AI interface in the chat() method:
- Changed from getImages() to getFiles()
- Added handling for documents alongside images
- Both sent as base64 data URIs

This change just makes the provider work with the standard Drupal AI interface that other providers (like OpenAI) also use, so files can be attached via ChatMessage::setFile()

I've still kept the file management methods (uploadFile, getFileUrl, listFiles, etc.) for any custom/advanced use cases.

Also fixed: Uninitialized $pages property error in responseToArray().

fago’s picture

Title: Add file upload » Support attaching images and files in chats, file uploads
Version: 1.0.x-dev » 1.1.x-dev

looks good, this is the really valuable feature - support for attaching files in chats!

MR needs a re-roll though.

fago’s picture

Status: Needs review » Needs work
petar_basic’s picture

Status: Needs work » Needs review

Updated MR & fixed phpstan issues

  • fago committed a7ad5c6f on 1.1.x authored by petar_basic
    feat: #3515048 Support attaching images and files in chats, file uploads...
fago’s picture

Status: Needs review » Fixed

all good. I had resolve conflicts, merged.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.