AI proving ground

AI Proving Ground is a pinned, reproducible Drupal site with known
contents, built to be the target AI agents are evaluated against. You point
an agent at a real Drupal, let it build, and grade what it actually built:
the saved configuration, the executed view, the persisted Canvas tree. Not
the text the model produced about its work.

Most AI-agent evals for Drupal run in an empty folder, so they grade what
a model says rather than what a build does. A plausible answer scores well
even when the site it describes would not work. The fix is a shared test
bed with three properties: known state (every graded fact is pinned and
verifiable), cheap reset (every run starts from the same baseline), and one
grader serving both the in-site eval runtime and external harnesses. This
project is that bed.

How it works

The bed is a base recipe (Canvas plus the eval stack) and a library of
fixture recipes. Each fixture seeds one coherent content and config set with
fixed UUIDs and declares exactly what it owns: config name prefixes and a
content-UUID namespace. An eval target names the fixtures it needs, pinned
as name@version; the bed composes them and refuses sets whose
ownership claims collide.

Four DDEV commands run the loop.
ai-eval-provision applies the base and the named fixtures, then
captures a baseline (database dump plus git ref) keyed by a content hash of
the exact fixture set, so the same set is a no-op and a changed set is a new
baseline. ai-eval-reset restores that baseline in seconds.
ai-eval-doctor verifies live state against the composed
manifest (module set, entity counts, every declared UUID resolving with its
declared label) and diagnoses drift precisely.
ai-eval-grade scores one task against live state and prints a
portable every_eval_ever 0.2.2 result envelope.

Grading is the deterministic droost_state oracle from
ai_eval_droost, the Droost-backed grader submodule of AI Eval:
assertions run in two tiers, the saved config shape and the actually
executed artifact, because a build can be config-valid and still
functionally wrong. The AI Eval runtime calls the
same grader in-process; an external harness such as Inspect AI calls it
over drush and gets the envelope. One grader, two directions, comparable
scores.

What ships

The base recipe; two fixtures (a small knowledge base of 12 pinned nodes
with a sectioned field, and the front page as a Canvas page entity) backing
a first dataset of build tasks; the four commands; the export tooling that
captures fixtures from a source site instead of hand-writing them; and an
agent skill documenting how to add fixtures, graded tasks, and target
bindings, so both people and coding agents can extend the bed the same
way.

Status and honesty

Early. The full provision, reset, and grade loop has been executed end to
end on a fresh unattended install (Drupal CMS via One Line
Installer), and the first real run surfaced five defects that are fixed
and documented in the repository, evidence included. Result envelopes
validate against the reference every_eval_ever models. Landed since
registration: the eval-target fixture binding shipped upstream in AI Eval,
so a target can pin the exact fixture set it expects, and ai_eval_droost is
now published as an AI Eval submodule. Not done yet: UUID-keyed row
assertions are pending in the AI Eval queue (the shipped dataset still pins
two install-specific node IDs, which fail on a fresh install), unattended
agent runs still need the planned integrity layer (baseline escrow and
tamper-evident grading), and there is no run-the-agent command; a reference
driver is under design in the issue queue. The repository's docs directory
carries the verification evidence for every claim on this page.

Requires Drupal 11.3 or later, AI Eval, Droost (with its
views and canvas submodules), and the ai_eval_droost submodule, which now
ships inside AI Eval: on its development branch today, and in tagged
releases from the next release after 1.0.0-beta1. Drupal CMS provides
Canvas and the default theme the shipped fixtures expect.