Overview

#3522013: External AI Chatbot Functionality Implemented the AI feature in Experience Builder, which currently supports only textual prompts to generate components. This issue aims to enhance the functionality by adding support for image uploads to help generate the desired component.

Proposed resolution

User interface changes

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

narendrar created an issue. See original summary.

narendrar’s picture

Component: Page builder » AI
narendrar’s picture

Title: [PP-1] Image upload feature in AI Chatbot » Image upload feature in AI Chatbot
narendrar’s picture

utkarsh_33 made their first commit to this issue’s fork.

utkarsh_33’s picture

Assigned: Unassigned » utkarsh_33

I'll assign it to me as this still needs some more work.

lauriii’s picture

StatusFileSize
new156.84 KB
new213.11 KB

Here are few examples that we could use to test this:

Prompt 1:

I want to build a hero component with heading, text and a slot for buttons. The highlighted part of the heading should use bold from CKEditor.

Prompt 2:
I want to build a hero component with heading, text and slot for buttons. There's also an image which is displayed on the right on desktop and hidden on mobile.

marcus_johansson’s picture

I've added som comments - most is just NIT and doesn't actually stop merging.

The bigger parts more UX questions, that I think laurii should answer if they are needed.

utkarsh_33’s picture

Thanks for the review @Marcus_Johansson. I have made the changes which does not require inputs from @laurii.Remaining things can be done once we get the confirmation from @laurii.

utkarsh_33’s picture

StatusFileSize
new2.19 MB
new121.77 KB

The current state of the MR works fine for creating simple code components. but there are some issues when there is an image which are as follows:-

  • When a user uploads an image as mentioned in #8's prompt 2 It makes it hard for the ai to generate exact similar doodle. It identifies that a doodle is present but does not generate one which is exact similar.
  • When a user uploads an image like this then it does insert an image which is close to this(in this case it inserted and image of forest without the human)

It does create simple components such as this one very well. Also as discussed with @laurii that if we tell the AI bot to create a component and also which image to insert in the images section then it does take care of that as well.

@Marcus_Johansson, @laurii we can discuss further what all needs to be taken care of in this issue.

tim.plunkett’s picture

We have four options:

  1. Do nothing, let them overspend
  2. Client-side validation preventing large uploads
  3. Client-side modal informing them it's too big
  4. Convert/downscale the image ourselves before passing to AI
utkarsh_33’s picture

I have created a follow-up issue which enables the client side validation of size for image uploads #3535108: Add client-side validation to prevent large (expensive) image uploads to LLM.

utkarsh_33’s picture

StatusFileSize
new39.24 KB

Uploading the SS showing how the image upload icon looks like.

mred9 made their first commit to this issue’s fork.

utkarsh_33’s picture

Status: Active » Needs review

I have merged all the new changes and also tested that no existing scenario is breaking and the new once are working as expected. Marking this NR.

narendrar’s picture

Status: Needs review » Needs work
utkarsh_33’s picture

Status: Needs work » Needs review

Addressed feedbacks.

narendrar’s picture

Status: Needs review » Needs work
StatusFileSize
new34.38 KB

Issue On attaching image, it seems to be hidden under help text.

I tested it manually and similar component gets created but it it has image, it is not replaced by a placeholder image as image url is wrong (not sure if this is an issue or not).

tim.plunkett’s picture

Status: Needs work » Needs review
narendrar’s picture

Status: Needs review » Reviewed & tested by the community

Created #3535530: XB AI: Image not displaying actual URL in generated component for #19 image issue.
Other changes looks good to me. Moving it to RTBC.

tim.plunkett’s picture

Status: Reviewed & tested by the community » Fixed

Merged!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

kristen pol’s picture

Assigned: utkarsh_33 » Unassigned
neha_bawankar’s picture

StatusFileSize
new146.34 KB

Tested changes on branch 1.x :

Scenario

Result

Status

Verify presence and clickability of Image Upload button Image Upload button is present and clickable PASS
Image preview after upload Image preview displayed after upload PASS
Upload a .jpeg image without a prompt No prompt provided PASS
Upload a .jpg image with prompt: create a matching code component Able to read image and create similar code component PASS
Upload a .png image with prompt: duplicate the format in code component Able to read image and create similar code component PASS
Upload a .gif image with prompt: create a matching code component Unable to select .gif file PASS
Upload a .mov file with prompt: create a matching code component Unable to select .mov file PASS
Upload a .heic file with prompt: create a matching code component Unable to select .heic file PASS
Upload a .svg file with prompt: create a matching code component Unable to select .svg file PASS
Upload image file of size 100 MB with a prompt tc04 FAIL
Replace uploaded image:

1. Add image and write a prompt

2. Remove image

3. Add new image and submit
Code component created in reference to new uploaded image PASS
Remove uploaded image:

1. Add image and write a prompt

2. Remove image and submit
Prompt submitted without image PASS
Drag-and-drop image upload Image uploaded, and typed prompt remains as is PASS
Upload multiple image files in one message Only one image is uploaded and processed PASS

Reported :