Overview

In #3539554: Add Playwright tests for XB AI module, we introduced Playwright tests for the Canvas AI module. However, there are still several scenarios that have not yet been covered. This issue aims to expand the test suite to ensure broader coverage.

Potential scenarios to cover:

  • Page builder task
  • Page history
  • Test current_layout, selected_component value gets updated
  • Also test selected_component value in above test
  • All tests for node (article)

(and add additional scenarios as needed)

Proposed resolution

User interface changes

Issue fork canvas-3542219

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

kunal.sachdev created an issue. See original summary.

narendrar’s picture

Issue summary: View changes
narendrar’s picture

Issue summary: View changes
jessebaker’s picture

Over in #3539554: Add Playwright tests for XB AI module (in MR 1466) the test "Should verify AI panel API request and response payload" was removed in an effort to reduce flakiness.

I'm dropping this comment here to acknowledge that it probably needs to be reinstated as part of this ticket in some form. However @justafish mentioned in chat that "I've commented out the API endpoint test as I don't think this is the appropriate place to test the structure of an API response" so perhaps a different approach would make sense.

jvbrian made their first commit to this issue’s fork.

narendrar’s picture

narendrar’s picture

Cover #3545378: Canvas AI: Agents fail due to missing entity type and entity ID parameters in this issue. I believe that 'Should verify AI panel API request and response payload' could have prevented this problem.

Project: Experience Builder » Drupal Canvas

Experience Builder has been renamed to Drupal Canvas in preparation for its beta release. You can now track issues on the new project page.

jvbrian’s picture

StatusFileSize
new3.62 MB

I've added a Playwright test to cover component generation via the AI chat interface. The test confirms that:

- The AI panel is accessible.
- Users can input a prompt (e.g., "Create a hero banner component with a title and button").
- The AI generates content in response.
- The generated component appears in the editor.

Attachment: video demostration.

Test file: `test-generate-component-with-ai.spec.ts`

This contributes to expanding test coverage for the XB AI module as outlined in the issue.

ankitv18’s picture

Issue summary: View changes
jvbrian’s picture

Assigned: Unassigned » jvbrian

I have created a new issue fork because Experience Builder has been renamed to Drupal Canvas.

jvbrian’s picture

StatusFileSize
new4.84 MB

This update extends test coverage by adding validation for page creation through the AI panel in Canvas.
The test now simulates user interaction with the AI panel and verifies successful page generation.
See attached video for visual proof of functionality.
Ready for review!

jvbrian’s picture

StatusFileSize
new8.82 MB

This update adds a Playwright test to validate Canvas state tracking during AI-driven component workflows.
The test covers component creation via the AI panel, addition to the component library, and insertion into the Canvas—verifying that current_layout and selected_component are correctly updated throughout.
See attached video for a full demonstration of the test in action.
Ready for review!

narendrar’s picture

Status: Active » Needs work

Can you please create a merge request for this? This will automatically run the created test and make it easier to review.
Also, I think these tests should not interact with the LLM directly; we should mock the response instead.
See:- tests/src/Playwright/ai.spec.ts

jvbrian’s picture

StatusFileSize
new3.25 MB

I have applied a fix to the CanvasEditor.spec.ts file because the ai.spec.ts test was throwing some errors, the settings are as follows:
- Wait for the editor UI.
- Detect the panel by stable data-testid instead of a “Library” heading.
- Do not click again if the panel is already open.
- Overlay reflects new instance count

jvbrian’s picture

StatusFileSize
new4.84 MB

2 new tests are added to cover these scenarios:
- Page builder task
- Page history.

pageBuilderTask.spec.ts
Tests of basic AI functionalities in the page builder: answering questions, creating components, generating metadata and titles, and processing editing commands.

pageHistory.spec.ts
AI History Persistence Tests: Verify that interactions are maintained across navigation, multiple queries, and page reloads.

Both use fixtures from the canvas_ai_test module to simulate AI responses in a predictable way.

jvbrian’s picture

StatusFileSize
new6.06 MB

The following tests are added to cover the remaining test cases:

currentLayoutSelectedComponent.spec.ts
nodeArticle.spec.ts
These tests cover the mentioned scenarios:

Test that the current_layout, selected_component value gets updated
Also test the selected_component value in the above test
All tests for node (article)
Additionally, alternative tests have been added to simulate other scenarios. A multimedia file is included demonstrating the tests in action.

jvbrian’s picture

Status: Needs work » Needs review
rakhimandhania’s picture

rakhimandhania’s picture

Issue tags: +AI Page Generation
jibran’s picture

Assigned: jvbrian » Unassigned
Status: Needs review » Needs work

Let's rebase this branch first and then mark this issue NR and unassign the issue so that it can be picked up for a review.

ajv009’s picture

Assigned: Unassigned » ajv009
ajv009’s picture

Assigned: ajv009 » Unassigned
Status: Needs work » Needs review

Hey @jibran, rebased the branch on current 1.x as you suggested.

The original test files couldn't be rebased cleanly because the Playwright infrastructure changed significantly upstream since the MR was first opened: CanvasEditor.ts was replaced by the Canvas mixin system, the test fixture imports changed to isolatedPerTest/parallelWorker, and tests moved into PlaywrightJail/ and tests/isolatedPerTest/. So I rewrote the tests from scratch using the current patterns.

@jvbrian, thank you for the thorough work on the original tests. A number of the scenarios you wrote (create component, generate title, generate metadata, edit component) are already covered by the existing ai.spec.ts in PlaywrightJail/, so I trimmed those to avoid duplication and kept the genuinely new coverage:

  • aiPageBuilderTask.spec.ts: AI question answering ("What is a CMS" with response text verification), and metadata + title generation in a single session.
  • aiHistory.spec.ts: Chat history persistence after page reload, after navigating away and back, multiple sequential queries in one session, and component creation flow with return to editor.

Both files use the canvas_ai_test module fixtures to mock LLM responses (per @narendrar's feedback in comment #14), and follow the same isolatedPerTest pattern as the existing ai.spec.ts.

Setting to "Needs review" and unassigning per your request. Happy to adjust if the scope should be different or if any of the dropped scenarios should be added back.

Transparency note: This rebase and rewrite were done with highly supervised usage of Claude Code. Not here to waste anyone's efforts or demean the work being done, and definitely not posting AI slop. I verified everything personally. If I'm not doing something right process-wise, please flag it, would be more than happy to course correct.

jibran’s picture

Assigned: Unassigned » jibran

Having a look at the MR.

jibran’s picture

Why are we not adding these tests in tests/src/Playwright?

ajv009’s picture

guess what. that was my GenAI workflow mistake now, hehe my bad @jibran...
do you mind if I take it back to tune my workflow,
I missed thinking about this as well when ai said, most of tests are already in playwright and so on,
lately I have been feeling am thinking less and less over time

jibran’s picture

Assigned: jibran » Unassigned
Status: Needs review » Needs work

Sure.

ajv009’s picture

Status: Needs work » Needs review

I have tuned my workflows a bit, hopefully this wont happen again

jibran’s picture

Status: Needs review » Needs work

aiHistory.spec.ts is failing on the CI.

ajv009’s picture

Status: Needs work » Needs review