AI proving ground

AI Proving Ground gives you a Drupal site with known content, made to test AI agents. You give an agent a task, for example "Make a View that shows these pages". The agent does the task on a real Drupal site. Then AI Proving Ground examines what the agent saved, and runs it. It does not use the report of the agent.

Why

An agent can report "done" when the work is wrong. For example, a View can have a valid configuration and show the wrong rows.

To find these errors, you need three things:

• A site with known content.
• A method to return the site to the same start state before each attempt.
• A check of the real result on the site.

This project gives you these three things as a Drupal recipe and some DDEV commands.

Who it is for

• Maintainers of modules and recipes who want to know if AI agents use their module correctly.
• Teams that compare agents, models or prompts on the same Drupal tasks.
• People who write evaluations with AI Eval and need a real site for them.

How it works

The grader is the code that does the checks. The Droost grader for Views and Canvas is part of AI Eval. AI Proving Ground also runs command checks: drush or PHP commands that check any saved state.

1. Prepare the site. ddev ai-eval-provision kb_docs canvas_home installs the sample content. Then it saves a snapshot: the database, the public files, and a git tag of the code.
2. Reset the site. ddev ai-eval-reset restores the database, the public uploaded files and the folders config/sync, web/modules/custom and recipes. Do a reset before each attempt.
3. Run the agent. ddev ai-eval-run dv01_view_build shows the task and records the attempt. Give the task to the agent that you want to test. When the agent stops, push Enter. The command then shows the grade command for this attempt.
4. Grade the result. ddev ai-eval-grade dv01_view_build --run=RUN_ID examines the site and shows the score as JSON. The run ID connects the score to the attempt.

ddev ai-eval-doctor examines the sample content, the necessary modules and the grader code. It tells you what changed.

An example

The repository has one task with two results. The second View has a default value on its exposed section filter. This configuration is valid, but the page shows 4 rows, not 12.

A configuration check accepts the two results. AI Proving Ground also runs the View. It finds that 8 rows are not on the page. The first result gets 5 points out of 5. The second result gets 0 points. The files and a script to do the test again are in docs/experiments/.

What you get

• Two sets of sample content: a small knowledge base of 12 pages with a section field, and a home page made with Canvas.
• Two sample tasks: make a View of the knowledge base, and add a heading to the Canvas home page.
• Command checks. A task can have a rubric with commands, for example drush or PHP, that check any saved state. AI Proving Ground runs them in the DDEV container.
• A score for each attempt, as JSON in the Every Eval Ever format. This is a shared format for evaluation results.
• The JSON result records the score, the model that you name, the AI Eval version, and hashes of the task and the grader code. Other people can compare these hashes with their copy of the task and the grader code.
• The score also goes into the AI Eval database. You can see it on the AI Eval results page.

Safety checks

These checks are for attempts that run without a person who watches.

• Reset and doctor tell you if the grader code or a module version changed after the snapshot.
• Rubrics are content on the site, so an agent can change them. The snapshot records each rubric. Before grading, AI Proving Ground checks that the task's rubric matches the snapshot. If it does not match, grading stops.
• You can keep a copy of the snapshot outside the project. After each attempt, run ddev ai-eval-verify kb_docs canvas_home. This command compares the snapshot and the grader code with the copy. It can restore the snapshot. It does not restore code. It only reports changed code.
• Some agents need credentials. The run command examines the DDEV configuration for credentials that the site shares with your computer. If it finds them, it does not start these agents. You can override this rule.

Your own tests

Use the two sample tasks as examples. To test your own module, do these steps:

1. Add your module. Require it with Composer, or copy it to web/modules/custom, before you make the snapshot. Reset removes a module that you add after the snapshot.
2. Make your content. Make the content and configuration for your test on a site. Export them with one command to recipes/fixture_NAME/ in your project. This folder is a fixture: a Drupal recipe with configuration and content. The content keeps its UUIDs after each import. Keep the fixture in the git repository of your project. Updates to AI Proving Ground do not change it.
3. Write your tasks. Write each task and its checks in a YAML file in the datasets/ folder of your module. The module must be enabled. To check changes to other parts of the site, add a rubric with command checks. Make the rubric before you make the snapshot.
4. Select the fixtures. Give the names of the fixtures for each test: ddev ai-eval-provision kb_docs NAME, where NAME is the name of your fixture. Each set of fixtures has its own snapshot.

The README shows all the steps.

Agent skills

The repository has two agent skills in the skills/ folder:

• running-proving-ground-evals: how to run and grade an attempt, and what each error message means.
• extending-the-proving-ground: how to add fixtures and tasks.

CAUTION: Give these skills only to the agent that runs the evaluations. Do not put them in the project of the agent that you test. The skills tell where the correct answers are.

Status

Alpha. On 4 November 2026, I tested 1.0.0-alpha4 on a Drupal CMS site that I upgraded from an earlier release: provision, doctor, the View task, command checks and the rubric check. The last test of all the steps on this page on a new site was on 16 October 2026, with 1.0.0-alpha1.

Known limits:

• There are only two sample tasks.
• The tasks and their correct answers are on the site. An agent can read them.
• One attempt gives one score. Do each task many times before you trust a result.
• AI Proving Ground needs DDEV.
• The built-in grader reads Views and Canvas pages. Command checks can grade other changes, but you must write them.
• On my test site, a reset took about 30 seconds.
• There is no supported way yet to give a tested module its API keys. The snapshot keeps the database and settings.php.

Requirements

• Drupal 11.3 or later, with Canvas. Drupal CMS includes Canvas.
• AI Eval 1.1.0-alpha6 or later. It includes the Droost grader and finds the sample tasks.
• Droost 2.0 or later.
• Tool API 1.0.0-beta12 or later.
• DDEV.

Installation

1. CAUTION: Include the option --agents none in the next command. Without this option, the installer adds Claude Code to the site, and your own Claude login goes into the site. An agent that you test must not have your login.
2. Make a new Drupal CMS site with Droost. Use the One Line Installer:
   bash <(curl -fsSL https://aibp.drupalstarforge.ai/install.sh) --yolo --name my-test-site --extras droost --agents none
3. Add the modules. The installer adds Droost 1, so give the versions: ddev composer require -W 'drupal/ai_eval:^1.1.0-alpha6@alpha' drupal/ai_proving_ground drupal/droost:^2.0 'drupal/tool:^1.0@beta'. Then run ddev drush updb -y.
4. Copy the DDEV commands from recipes/ai_proving_ground/ddev/ to the .ddev/ folder of your project. The README shows the commands to do this.
5. Remove the line /recipes from .gitignore. The snapshot needs the recipes folder in git.
6. Make a git repository for the project.
7. Commit all the files.
8. Run ddev ai-eval-provision kb_docs canvas_home.
9. Run ddev ai-eval-doctor kb_docs canvas_home.

Send questions, bugs and ideas to the issue queue.

AI Proving Ground gives you a Drupal site with known content, made to test AI agents. You give an agent a task, for example "Make a View that shows these pages". The agent does the task on a real Drupal site. Then AI Proving Ground examines what the agent saved, and runs it. It does not use the report of the agent.

Why

An agent can report "done" when the work is wrong. For example, a View can have a valid configuration and show the wrong rows.

To find these errors, you need three things:

• A site with known content.
• A method to return the site to the same start state before each attempt.
• A check of the real result on the site.

This project gives you these three things as a Drupal recipe and some DDEV commands.

Who it is for

• Maintainers of modules and recipes who want to know if AI agents use their module correctly.
• Teams that compare agents, models or prompts on the same Drupal tasks.
• People who write evaluations with AI Eval and need a real site for them.

How it works

The grader is the code that does the checks. The Droost grader for Views and Canvas is part of AI Eval. AI Proving Ground also runs command checks: drush or PHP commands that check any saved state.

1. Prepare the site. ddev ai-eval-provision kb_docs canvas_home installs the sample content. Then it saves a snapshot: the database, the public files, and a git tag of the code.
2. Reset the site. ddev ai-eval-reset restores the database, the public uploaded files and the folders config/sync, web/modules/custom and recipes. Do a reset before each attempt.
3. Run the agent. ddev ai-eval-run dv01_view_build shows the task and records the attempt. Give the task to the agent that you want to test. When the agent stops, push Enter. The command then shows the grade command for this attempt.
4. Grade the result. ddev ai-eval-grade dv01_view_build --run=RUN_ID examines the site and shows the score as JSON. The run ID connects the score to the attempt.

ddev ai-eval-doctor examines the sample content, the necessary modules and the grader code. It tells you what changed.

An example

The repository has one task with two results. The second View has a default value on its exposed section filter. This configuration is valid, but the page shows 4 rows, not 12.

A configuration check accepts the two results. AI Proving Ground also runs the View. It finds that 8 rows are not on the page. The first result gets 5 points out of 5. The second result gets 0 points. The files and a script to do the test again are in docs/experiments/.

What you get

• Two sets of sample content: a small knowledge base of 12 pages with a section field, and a home page made with Canvas.
• Two sample tasks: make a View of the knowledge base, and add a heading to the Canvas home page.
• Command checks. A task can have a rubric with commands, for example drush or PHP, that check any saved state. AI Proving Ground runs them in the DDEV container.
• A score for each attempt, as JSON in the Every Eval Ever format. This is a shared format for evaluation results.
• The JSON result records the score, the model that you name, the AI Eval version, and hashes of the task and the grader code. Other people can compare these hashes with their copy of the task and the grader code.
• The score also goes into the AI Eval database. You can see it on the AI Eval results page.

Safety checks

These checks are for attempts that run without a person who watches.

• Reset and doctor tell you if the grader code or a module version changed after the snapshot.
• Rubrics are content on the site, so an agent can change them. The snapshot records each rubric. Before grading, AI Proving Ground checks that the task's rubric matches the snapshot. If it does not match, grading stops.
• You can keep a copy of the snapshot outside the project. After each attempt, run ddev ai-eval-verify kb_docs canvas_home. This command compares the snapshot and the grader code with the copy. It can restore the snapshot. It does not restore code. It only reports changed code.
• Some agents need credentials. The run command examines the DDEV configuration for credentials that the site shares with your computer. If it finds them, it does not start these agents. You can override this rule.

Your own tests

Use the two sample tasks as examples. To test your own module, do these steps:

1. Add your module. Require it with Composer, or copy it to web/modules/custom, before you make the snapshot. Reset removes a module that you add after the snapshot.
2. Make your content. Make the content and configuration for your test on a site. Export them with one command to recipes/fixture_NAME/ in your project. This folder is a fixture: a Drupal recipe with configuration and content. The content keeps its UUIDs after each import. Keep the fixture in the git repository of your project. Updates to AI Proving Ground do not change it.
3. Write your tasks. Write each task and its checks in a YAML file in the datasets/ folder of your module. The module must be enabled. To check changes to other parts of the site, add a rubric with command checks. Make the rubric before you make the snapshot.
4. Select the fixtures. Give the names of the fixtures for each test: ddev ai-eval-provision kb_docs NAME, where NAME is the name of your fixture. Each set of fixtures has its own snapshot.

The README shows all the steps.

Agent skills

The repository has two agent skills in the skills/ folder:

• running-proving-ground-evals: how to run and grade an attempt, and what each error message means.
• extending-the-proving-ground: how to add fixtures and tasks.

CAUTION: Give these skills only to the agent that runs the evaluations. Do not put them in the project of the agent that you test. The skills tell where the correct answers are.

Status

Alpha. On 4 November 2026, I tested 1.0.0-alpha4 on a Drupal CMS site that I upgraded from an earlier release: provision, doctor, the View task, command checks and the rubric check. The last test of all the steps on this page on a new site was on 16 October 2026, with 1.0.0-alpha1.

Known limits:

• There are only two sample tasks.
• The tasks and their correct answers are on the site. An agent can read them.
• One attempt gives one score. Do each task many times before you trust a result.
• AI Proving Ground needs DDEV.
• The built-in grader reads Views and Canvas pages. Command checks can grade other changes, but you must write them.
• On my test site, a reset took about 30 seconds.
• There is no supported way yet to give a tested module its API keys. The snapshot keeps the database and settings.php.

Requirements

• Drupal 11.3 or later, with Canvas. Drupal CMS includes Canvas.
• AI Eval 1.1.0-alpha6 or later. It includes the Droost grader and finds the sample tasks.
• Droost 2.0 or later.
• Tool API 1.0.0-beta12 or later.
• DDEV.

Installation

1. CAUTION: Include the option --agents none in the next command. Without this option, the installer adds Claude Code to the site, and your own Claude login goes into the site. An agent that you test must not have your login.
2. Make a new Drupal CMS site with Droost. Use the One Line Installer:
   bash <(curl -fsSL https://aibp.drupalstarforge.ai/install.sh) --yolo --name my-test-site --extras droost --agents none
3. Add the modules. The installer adds Droost 1, so give the versions: ddev composer require -W 'drupal/ai_eval:^1.1.0-alpha6@alpha' drupal/ai_proving_ground drupal/droost:^2.0 'drupal/tool:^1.0@beta'. Then run ddev drush updb -y.
4. Copy the DDEV commands from recipes/ai_proving_ground/ddev/ to the .ddev/ folder of your project. The README shows the commands to do this.
5. Remove the line /recipes from .gitignore. The snapshot needs the recipes folder in git.
6. Make a git repository for the project.
7. Commit all the files.
8. Run ddev ai-eval-provision kb_docs canvas_home.
9. Run ddev ai-eval-doctor kb_docs canvas_home.

Send questions, bugs and ideas to the issue queue.