AI proving ground

AI Proving Ground is a pinned, reproducible Drupal site with known
contents, built to be the target AI agents are evaluated against. You point
an agent at a real Drupal, let it build, and grade what it actually built:
the saved configuration, the executed view, the persisted Canvas tree. Not
the text the model produced about its work.

Most AI-agent evals for Drupal run in an empty folder, so they grade what
a model says rather than what a build does. A plausible answer scores well
even when the site it describes would not work. The fix is a shared test
bed with three properties: known state (every graded fact is pinned and
verifiable), cheap reset (every run starts from the same baseline), and one
grader serving both the in-site eval runtime and external harnesses. This
project is that bed.

How it works

The bed is a base recipe (Canvas plus the eval stack) and a library of
fixture recipes. Each fixture seeds one coherent content and config set with
fixed UUIDs and declares exactly what it owns: config name prefixes and a
content-UUID namespace. An eval target names the fixtures it needs, pinned
as name@version; the bed composes them and refuses sets whose
ownership claims collide.

Four DDEV commands run the loop.
ai-eval-provision applies the base and the named fixtures, then
captures a baseline (database dump plus git ref) keyed by a content hash of
the exact fixture set, so the same set is a no-op and a changed set is a new
baseline. ai-eval-reset restores that baseline in seconds.
ai-eval-doctor verifies live state against the composed
manifest (module set, entity counts, every declared UUID resolving with its
declared label) and diagnoses drift precisely.
ai-eval-grade scores one task against live state and prints a
portable every_eval_ever 0.2.2 result envelope.

Grading is the deterministic droost_state oracle from
ai_eval_droost, the Droost-backed grader submodule of AI Eval:
assertions run in two tiers, the saved config shape and the actually
executed artifact, because a build can be config-valid and still
functionally wrong. The AI Eval runtime calls the
same grader in-process; an external harness such as Inspect AI calls it
over drush and gets the envelope. One grader, two directions, comparable
scores.

What ships

The base recipe; two fixtures (a small knowledge base of 12 pinned nodes
with a sectioned field, and the front page as a Canvas page entity) backing
a first dataset of build tasks; the four commands; the export tooling that
captures fixtures from a source site instead of hand-writing them; and an
agent skill documenting how to add fixtures, graded tasks, and target
bindings, so both people and coding agents can extend the bed the same
way.

Status and honesty

Early. The full provision, reset, and grade loop has been executed end to
end on a fresh unattended install (Drupal CMS via One Line
Installer), and the first real run surfaced five defects that are fixed
and documented in the repository, evidence included. Result envelopes
validate against the reference every_eval_ever models. Landed since
registration: the eval-target fixture binding shipped upstream in AI Eval,
so a target can pin the exact fixture set it expects, and ai_eval_droost is
now published as an AI Eval submodule (in tagged releases since
1.0.0-beta2). UUID-keyed row assertions shipped in AI Eval 1.0.0-beta3
(#3594736):
the dataset keys ground truth on fixture UUIDs, which survive
re-provisioning, where install-specific node IDs do not, so the shipped
dataset grades any provision of its fixtures with zero hand edits
(verified 2026-08-15 on a bed provisioned from scratch on released
versions only). A worked graded example ships in the repository under
docs/experiments/: the same build twice, one line apart, where
the defective build passes every configuration check and only executing
the view catches it, with the result envelopes, provenance hashes, and a
copy-paste reproduce script included. A reference run-the-agent command
(ddev ai-eval-run) shipped in
#3611932;
it drives nothing by default and refuses adapters that would hand an agent
the host's credentials. Not done yet: unattended agent runs still need the
planned integrity layer (baseline escrow and tamper-evident grading; the
reset and staleness gaps are filed as
#3611933
and #3611934).
The repository's docs directory carries the verification evidence for
every claim on this page.

Installation

Requires Drupal 11.3 or later, AI Eval
(1.0.0-beta2 or later; it ships the ai_eval_droost grader submodule) and
Droost (with its views
and canvas submodules). Drupal CMS provides Canvas and the default theme
the shipped fixtures expect.

composer require drupal/ai_eval drupal/ai_proving_ground
brings in the scoring layer and this recipe. Then copy the four commands
from the recipe's ddev/commands/web/ into your project's
.ddev/commands/web/, make sure the project is a git tree
(baselines need a ref), and run ddev ai-eval-provision. The
repository README walks the full loop, including the unattended
one-line-installer path.