Provide the AI Search Media Library as a tool/Agent [#3541872]

--- AI TRACKER METADATA ---
Update Summary: Provide the AI Search Media Library as a tool/Agent
Check-in Date: MM/DD/YYYY (US format) [When we should see progress/get an update]
Due Date: MM/DD/YYYY (US format) [When the issue should be fully completed]
Blocked by: [#XXXXXX] (New issues on new lines)
Additional Collaborators: @username1, @username2
AI Tracker found here: https://www.drupalstarforge.ai/
--- END METADATA ---

Overview

Detaills can be found here:
#3541825: [Meta] AI Image and Media Agents Track Assigned to: tonypaulbarker

TODO - Create rest of issue.
This should focus on using no semantic search and standard search processes for accessing the media library.
We don't know where this code should live. In XP? Or have another module? Media module?

Proposed resolution

User interface changes

Comments

Comment #1

18 August 2025 at 14:13

yautja_cetanu created an issue. See original summary.

Comment #2

catia_penas commented 8 September 2025 at 11:11

Issue tags:

+strategic evolution

Comment #3

catia_penas commented 8 September 2025 at 11:34

Issue summary:

View changes

Comment #4

8 September 2025 at 11:34

Project:

Experience Builder

» Drupal Canvas

Experience Builder has been renamed to Drupal Canvas in preparation for its beta release. You can now track issues on the new project page.

Comment #5

marcus_johansson commented 18 September 2025 at 09:12

Demoing

So for anyone that wants to test this with a RAG database in ddev, you can:

1. Add the following file in your .ddev directory: https://github.com/salsadigitalauorg/xb_metadata/blob/develop/.ddev/dock...
2. Add the following as a post-start hook in ddev: - exec: psql --user db --host pgvector -c "CREATE EXTENSION vector;"
3. Apply the following recipe: https://github.com/salsadigitalauorg/xb_metadata/tree/develop/recipes/xb...
4. Login and resave every media file
5. Attach the media finder agent wherever you need it and ask it to find images for you.

The fourth last step will:
1. Have a vision model look at the image and describe it very verbosely.
2. Index it.

A real solution

My suggestion for a real solution that works on multiple systems would be a function call that has configurations attached to it, where it can do graceful fallback.

So in the settings page of it you can, choose which kind of solution you want, which will include:
1. Search API RAG Search - you link it to a media index, which will be the best solution. This has multiple dependencies (AI, VDB provider, Search API, Search API AI, AI Provider)
2. Search API DB Search - you link it to a media index. This requires Search API.
3. Free text search with LIKE "%something%" or something similar - you link it to the media type.

On installation it will fallback to #3 and any recipe has to setup #2 or #1.

Comment #6

joshua1234511

English

Goa

commented 8 October 2025 at 05:22

Tested the solution
Configured a RAG database in DDEV with pgvector enabled and a pplied the xb_metadata recipe, linking Drupal media entities with AI embeddings. Resaved all media files to trigger the vision model, which generated detailed image descriptions and stored vector embeddings for search.

Issues with Integrating and tested the media finder agent. (Media finder agent works but with limited or inconsistent results.) Gemini AI

Other issues were
- Some images don’t get proper AI-generated descriptions.

Next steps - Review xb_metadata logs for model or API errors.

Comment #7

joshua1234511

English

Goa

commented 20 October 2025 at 02:07

Assigned:

Unassigned

» joshua1234511

Comment #8

yautja_cetanu commented 11 November 2025 at 14:14

First step is the classification: https://www.drupal.org/project/ai_recipe_image_classification

We then need:

AI Search tool recipe that uses this to allow AI to find the images. - Marcus TODO
It will need graceful fallback. So check RAG first and then if not use traditional search.
It isn't a simple fall back as AI will have to query the second in a different way knowing that it isn't semantic search

- This means Canvas AI might require 2 additional Recipies (Although tbh I think image classification and AI Search for images should be one recipe.)
- It will require Search API at the very least.

Arguments for a seperate recipe

There is a strong liklihood that this Image Search Recipe will be used outside of Canvas:

- People might just want to browse to images in their media library for merely managing their images and the search would be better with pure semantic search and no chatbots.
- Similarly people might want the chatbot to help them find images (lets say they are cleaning up the images, or downloading it for some other purpose).
- People who are using CKeditor and not canvas will likely want to be able to use AI to find images.
- People using the media field across Drupal will likely benefit from this.
- There are many other Agents outside canvas that may want an AI Search tool. (Maybe profiles? Creating items in a directory)
- If there is a single recipe, we can make the Image Search with AI function consistently across the use-cases and improvements in one will reflect the rest. End-users who want to fine tune how Agents find images (For example make use of custom metadata or use groups module for permissions to different collections of images), they have a single place to customise it.

I don't see why we'd want to seperate image classification from image search however other than a Drupal tendency to try and make things as atomic as possible.