Needs review
Project:
Drupal Canvas
Version:
1.x-dev
Component:
AI
Priority:
Normal
Category:
Task
Assigned:
Unassigned
Reporter:
Created:
20 Aug 2025 at 11:55 UTC
Updated:
6 Apr 2026 at 21:36 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #2
narendrarComment #3
narendrarComment #4
jessebaker commentedOver in #3539554: Add Playwright tests for XB AI module (in MR 1466) the test "Should verify AI panel API request and response payload" was removed in an effort to reduce flakiness.
I'm dropping this comment here to acknowledge that it probably needs to be reinstated as part of this ticket in some form. However @justafish mentioned in chat that "I've commented out the API endpoint test as I don't think this is the appropriate place to test the structure of an API response" so perhaps a different approach would make sense.
Comment #6
narendrarCover #3543786: Handle ParseException in XbBuilder Controller in this issue.
Comment #7
narendrarCover #3545378: Canvas AI: Agents fail due to missing entity type and entity ID parameters in this issue. I believe that 'Should verify AI panel API request and response payload' could have prevented this problem.
Comment #9
jvbrian commentedI've added a Playwright test to cover component generation via the AI chat interface. The test confirms that:
- The AI panel is accessible.
- Users can input a prompt (e.g., "Create a hero banner component with a title and button").
- The AI generates content in response.
- The generated component appears in the editor.
Attachment: video demostration.
Test file: `test-generate-component-with-ai.spec.ts`
This contributes to expanding test coverage for the XB AI module as outlined in the issue.
Comment #10
ankitv18 commentedComment #11
jvbrian commentedI have created a new issue fork because Experience Builder has been renamed to Drupal Canvas.
Comment #12
jvbrian commentedThis update extends test coverage by adding validation for page creation through the AI panel in Canvas.
The test now simulates user interaction with the AI panel and verifies successful page generation.
See attached video for visual proof of functionality.
Ready for review!
Comment #13
jvbrian commentedThis update adds a Playwright test to validate Canvas state tracking during AI-driven component workflows.
The test covers component creation via the AI panel, addition to the component library, and insertion into the Canvas—verifying that current_layout and selected_component are correctly updated throughout.
See attached video for a full demonstration of the test in action.
Ready for review!
Comment #14
narendrarCan you please create a merge request for this? This will automatically run the created test and make it easier to review.
Also, I think these tests should not interact with the LLM directly; we should mock the response instead.
See:- tests/src/Playwright/ai.spec.ts
Comment #15
jvbrian commentedI have applied a fix to the CanvasEditor.spec.ts file because the ai.spec.ts test was throwing some errors, the settings are as follows:
- Wait for the editor UI.
- Detect the panel by stable data-testid instead of a “Library” heading.
- Do not click again if the panel is already open.
- Overlay reflects new instance count
Comment #17
jvbrian commented2 new tests are added to cover these scenarios:
- Page builder task
- Page history.
pageBuilderTask.spec.ts
Tests of basic AI functionalities in the page builder: answering questions, creating components, generating metadata and titles, and processing editing commands.
pageHistory.spec.ts
AI History Persistence Tests: Verify that interactions are maintained across navigation, multiple queries, and page reloads.
Both use fixtures from the canvas_ai_test module to simulate AI responses in a predictable way.
Comment #18
jvbrian commentedThe following tests are added to cover the remaining test cases:
currentLayoutSelectedComponent.spec.ts
nodeArticle.spec.ts
These tests cover the mentioned scenarios:
Test that the current_layout, selected_component value gets updated
Also test the selected_component value in the above test
All tests for node (article)
Additionally, alternative tests have been added to simulate other scenarios. A multimedia file is included demonstrating the tests in action.
Comment #19
jvbrian commentedComment #20
rakhimandhania commentedComment #21
rakhimandhania commentedComment #22
jibranLet's rebase this branch first and then mark this issue NR and unassign the issue so that it can be picked up for a review.
Comment #23
ajv009 commentedComment #24
ajv009 commentedHey @jibran, rebased the branch on current 1.x as you suggested.
The original test files couldn't be rebased cleanly because the Playwright infrastructure changed significantly upstream since the MR was first opened:
CanvasEditor.tswas replaced by theCanvasmixin system, the test fixture imports changed toisolatedPerTest/parallelWorker, and tests moved intoPlaywrightJail/andtests/isolatedPerTest/. So I rewrote the tests from scratch using the current patterns.@jvbrian, thank you for the thorough work on the original tests. A number of the scenarios you wrote (create component, generate title, generate metadata, edit component) are already covered by the existing
ai.spec.tsinPlaywrightJail/, so I trimmed those to avoid duplication and kept the genuinely new coverage:aiPageBuilderTask.spec.ts: AI question answering ("What is a CMS" with response text verification), and metadata + title generation in a single session.aiHistory.spec.ts: Chat history persistence after page reload, after navigating away and back, multiple sequential queries in one session, and component creation flow with return to editor.Both files use the
canvas_ai_testmodule fixtures to mock LLM responses (per @narendrar's feedback in comment #14), and follow the sameisolatedPerTestpattern as the existingai.spec.ts.Setting to "Needs review" and unassigning per your request. Happy to adjust if the scope should be different or if any of the dropped scenarios should be added back.
Transparency note: This rebase and rewrite were done with highly supervised usage of Claude Code. Not here to waste anyone's efforts or demean the work being done, and definitely not posting AI slop. I verified everything personally. If I'm not doing something right process-wise, please flag it, would be more than happy to course correct.
Comment #25
jibranHaving a look at the MR.
Comment #26
jibranWhy are we not adding these tests in
tests/src/Playwright?Comment #27
ajv009 commentedguess what. that was my GenAI workflow mistake now, hehe my bad @jibran...
do you mind if I take it back to tune my workflow,
I missed thinking about this as well when ai said, most of tests are already in playwright and so on,
lately I have been feeling am thinking less and less over time
Comment #28
jibranSure.
Comment #29
ajv009 commentedI have tuned my workflows a bit, hopefully this wont happen again
Comment #30
jibranaiHistory.spec.ts is failing on the CI.
Comment #31
ajv009 commented