Problem/Motivation

Varbase AI Safety has no functional tests. Its CI only validates composer.json and the YAML files. Nothing checks that the recipe applies on a fresh Varbase 11 site, twice, and that editors, admins and visitors get what it promises.

Proposed resolution

  • Add the Varbase functional testing suite (Playwright and Cucumber through @vardot/varbase-e2e): the guardrail sets, AI logging kept 90 days, an editor image upload checked by the vision model and refused with the safety message, the check in the AI log, and who opens the guardrails and the log.
  • Add a varbase-e2e-test CI job: a fresh Varbase 11 site, the recipe applied twice, then the suite. It checks the apply stops without a default model, then uses the AI module test provider (EchoAI) and installs ImageMagick, so no AI key is needed.
  • Scenarios that need a real AI provider are tagged @ai-provider and run in their own job only when an OPENAI_API_KEY CI variable is set.

Remaining tasks

  • ✅ File an issue
  • ❌ Addition/Change/Update/Fix
  • ❌ Testing to ensure no regression
  • ❌ Automated unit/functional testing coverage
  • ❌ Developer Documentation support
  • ➖ User Guide Documentation support
  • ➖ UX/UI designer responsibilities
  • ➖ Accessibility and Readability
  • ❌ Reviewed by a human
  • ❌ Code review by maintainers
  • ❌ Full testing and approval
  • ❌ Credit contributors
  • ❌ Review with the product owner
  • ❌ Update Release Notes
  • ❌ Release

User interface changes

  • N/A

API changes

  • N/A

Data model changes

  • N/A

Release notes snippet

  • Added the Varbase functional testing suite and a CI job that runs it on a fresh Varbase 11 site.
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

rajab natshah created an issue. See original summary.

  • rajab natshah committed 903a7212 on 1.0.x
    test: #3633124 Test what users do on amazee.ai from Drupal CMS AI, in...
rajab natshah’s picture

Status: Active » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.