Visual Review and Iteration agent on created components

Follow up

• Need to turn it into a proper agent now file upload hasn't happened.
• Make it so we can swap out the back-end service for looking at the images.
• Make it so we can configure the number of loops
• UX Work - What does the user see whilst its doing its steps? (It has re-write the code for the componenent and re-write the output, will a user see it updating? Will it be greyed out so they can't make manual changes?)

Switching to the correct tag

Currently, the code component creation agent returns the component data (id, js, css etc) in JSON format and based on that, the actual component is created from the UI.
But to review the component on the fly before sending the final output, the component must exist. Means, we should create it directly from the backend.

As mentioned in #12, component creation is currently happening from the UI. For the review to work, the component must be created from the backend. However, in that case, the CSS and JS will not be compiled properly, since the compilation happens in the frontend and the compiled CSS and JS are then sent to the backend. Because of this, the component will not be rendered in the preview controller.

Also, when Chromium is used, the component is rendered as seen by an anonymous user. XB does not apply styles for anonymous users. The Create/Edit XB page content permission is required for this. Even if the permission is granted, the issue described above still prevents the component from being previewed.

Also, currently the component agent doesn't actually 'see' the component image. The image is given only to the orchestrator agent and orchestrator 'describes' the image to the component agent. I think the output of first iteration would be more similar to the uploaded image, if the image is actually given to the component agent. Few options that come to my mind are

• Create a tool for the component agent, get_image_content, that simply returns the output of file_get_contents($image_url).
• Subscribe to Drupal\ai\Event\PreGenerateResponseEvent to check if the orchestrator has an image input → store the image ID somewhere → then attach the image when the component agent is triggered, using the same event subscriber.

I tried option 2. The Drupal\ai\OperationType\GenericType\ImageFile class currently only contains the filename, mime type, and file binary data. Perhaps an issue could be created for the AI module to also include the file id for images.

For people not into AI basically we need to find a way that we automatically generate an accurate screenshot of a componenet.

- The componenet is built in react by typing javascript, css etc.
- We need some way we can take the componenet we see rendered and automatically take a screenshot without the user having to press screenshot screens.
- We need to send that screenshot to a server side API for processing (in this case an AI agent looking at it) whilst thinking carefully of any security implications.

Andrew has created a video of the new htmltocanvas pro:

xb-ai-canvas.mp4

It shows a very simple way of this working but its not perfect, you can see the colour of "New Introducing Framer forms" isn't quite right.

This with with the following option:

{
  foreignObjectRendering: true
}

I'm not sure the implication on browser support - this is the description from the docs:

Whether to use ForeignObject rendering if the browser supports it

Here is another approach using the screen capture API.

There is an issue of blur in this. That may not affect the LLM's looping as it may not care about image resolution/quality. However, I believe it is to do with DPI of the source vs the target canvas, which I expect is resolvable (perhaps this article would help).

I think we'd also want to look at finding a way to resize the iframe to the "correct" size to avoid scrolling etc, which the canvas approach bypasses.

Again some rough and ready code:

const iframe = document.querySelector('iframe[title="XB Code Editor Preview"]') as HTMLIFrameElement;

  let captureStream = null;

  try {
    captureStream = await navigator.mediaDevices.getDisplayMedia({
      video: {
        displaySurface: "browser",
      },
      audio: false,
      // @ts-ignore
      preferCurrentTab: true,
      selfBrowserSurface: "include",
      systemAudio: "exclude",
      surfaceSwitching: "include",
      monitorTypeSurfaces: "include",
    });

    const track = captureStream.getVideoTracks()[0];
    // @ts-ignore
    const imageCapture = new window.ImageCapture(track);
    const bitmap = await imageCapture.grabFrame();
    const canvas = document.createElement("canvas") as HTMLCanvasElement;
    canvas.width = iframe.offsetWidth;
    canvas.height = iframe.offsetHeight;
    const bounding = iframe.getBoundingClientRect();
    (canvas.getContext("2d") as CanvasRenderingContext2D).drawImage(bitmap, bounding.left, bounding.top, iframe.offsetWidth, iframe.offsetHeight, 0, 0, iframe.offsetWidth, iframe.offsetHeight);

    const img = document.createElement('img');
    img.src = canvas.toDataURL('image/png');
    img.style.position = 'absolute';
    img.style.top = '10px';
    img.style.left = '10px';
    img.style.border = '2px solid green';
    document.body.appendChild(img);
  } catch (err) {
    console.error(`Error: ${err}`);
  }

xb-ai-review-2.mp4 is a quick video of the change in merge request !1462.

At the very least it would need a better icon, and a review of my noddy React code. If we want to try and get this merged, it might want a feature flag for now. We were thinking that we might try and support a few different approaches, giving scope for server side headless browser where available as a more reliable and direct alternative. So a setting with options which allow us to control what we do, with a default of "disabled", might be a good approach.

I'm wondering if we're approaching images and implementing designs the right way. It doesn't look like even with the latest changes the design produced matches closely enough the original image.

I also don't think it makes sense that the image is visible in the chat. It's confusing to see a new image appear there that I didn't upload myself.

I'm also wondering if we should upload multiple images because some issues could happen on a specific breakpoint?

I'll bring this up at the XB AI meetup but I think we should create another module for this to experiment on, could be a sandbox module, could be something else and maybe bring it back in. As I think we need to find out how to make something that physically works before figuring out the ideal UI.

There are three approaches:

• Use the html2canvas stuff in the example above - This is the simplest, and provides high resolution screenshots, but they are sometimes inaccurate (incorrect fonts, background colours, etc).
• Use the browser screenshot tool - This requires the User to click a button giving permission for the screen to be shared with XB AI like when you share a screen in google meet. It provides accurate images but they are low resolution and grainy (which I think will be fine, AI doesn't seem to do better with higher res images as it takes up more tokens.
• php chrome or playwrite, server side with a headless browser - This is the most likely to work method, and will be necessary if we use agents working in the background so we want to do this eventually. But its the hardest to host and make work. (Agents need to direct a browser).

I think we need to make a module that allows us to switch between the 3 above so we can try prompt engineering to get it to work well. With testing a review agent using chatgpt playground, we got the design to look pretty good after one iteration. As you can see above, in the canvas one, it didn't look good after one iteration (but that might just be the prompt). When we have a better idea of what works we can think about the different aspects of the UI we have to worry about:

• Accuracy of the Image - One reason why we went for making it so it uploaded the screenshot in the chat is that the screenshot isn't accurate sometimes. This gives the end-user the ability to see and understand why its not quite working. (But obviously if we can make the screenshots accurate then we don't need this.
• How long this takes - Review agents significantly increase the cost and time it takes to do anything. We may need some UI that allows people to choose (Like chatgpt has thinking mode vs not) or can we make it happen quickly?
• Why are these review agents happening and what prompts should fire them off or not? In the demo above we side-stepped it by giving the user a button. But the example of, using a figma screenshot and making an exact duplicate is unlikely to be a major way the chatbot is used, if that is our goal a dedicated screenshot to drupal tool is probably better. Its more likely they will upload images for inspirtional. So that would change things a little. (Maybe you don't want a review agent as you want user feedback quickly).

So my proposal is:

We create a module that allows you to pick between 3 different approaches and we can try the prompt engineering to figure out what is the best approach and then we can use this information to answer the UX questions above (eg break points etc).