AI Eval

AI Eval helps you find out whether AI features on a Drupal site are getting better or worse. Describe a good answer, run a fixed set of questions, score each answer with judges and checks, then compare the result with a quality gate. It works in three modes. Agent mode tests an AI Agents plugin from end to end. Chat mode tests a provider, model, and prompt without an agent framework. Workflow mode (1.1.x) tests a feature that a site already runs, such as AI Answers, a stored AI prompt, a field automator, or a Search API index.

Start with a guided test

Create an agent or chat test in three steps. First choose the AI, graders, and quality gate. Next add examples or choose an existing dataset. Finally review the test, then save it, or save it and go straight to the run button. Drafts remain available for seven days. When several tests share a dataset, you can update the shared dataset or create a copy for one test.

Test a workflow (1.1.x)

Workflow mode runs a feature the way the site runs it, then grades what comes out. Each kind of feature has its own optional submodule. Enable only the ones you need. You define a workflow test in configuration, not in the guided form. The test's edit page shows its workflow settings but does not change them.

• AI Eval AI Answers runs an AI Answers agent, so you can grade the answers visitors get on search.
• AI Eval AI Prompt runs a stored AI prompt.
• AI Eval AI Automators runs one field automator on a content entity that is never saved.
• AI Eval Search API grades the ranked results from a Search API index: did the right page appear in the top results, and how high.

AI Eval Content Review Judge uses an AI Content Review criterion as a judge, so you can validate the criterion against pages labeled by people.

Read a result

Each result starts with “N of M examples met your expectations.” Questions that need attention appear first. Run history records completed, cancelled, errored, and interrupted runs. You can resume an interrupted run or start a new one.

Compare two runs of the same test. When both runs used the same questions, graders, judge, and quality gate, the page reports a direct comparison, even if the model or prompt under test changed. When one of those differed, it names what changed, so read the difference with care. When a run did not record its settings, the page shows the two results side by side without a verdict.

On 1.1.x a result also names the model under test and the judge models, and shows the tokens used per question and for the run. A test can mark a grader as must pass: when that grader scores a question below the pass threshold, the question fails, whatever the other graders say.

On 1.1.x a run does not start on a dataset with no questions to run, and says why. Deleting a dataset deletes its questions, and the confirmation page lists the tests that use it.

Know whether to trust the judge

Validate each LLM judge against answers labeled by people. AI Eval measures how often the judge agrees with people on good answers and on bad answers. A trust badge on the Dashboard shows the current state. Results warn you when a quality gate used a judge that has not passed validation. On 1.1.x you can also compare two judges on the same labeled answers, and see which answers a judge gets wrong every time.

On 1.1.x a judge is trusted only after at least 10 pass and 10 fail labels, with every labeled answer scored. Answers the judge did not score show as Not scored instead of counting as failures. Editing a judge's prompt marks its validation out of date.

Calibrate yes/no decisions (1.1.x)

Some AI features answer yes or no and give a probability that the answer is yes. A decision calibration checks those numbers against people: when the model says 80% likely, do people agree it is a yes about 8 times out of 10? People review a sample of the answers without seeing the model's probability. AI Eval builds a correction from the reviews. It keeps part of the answers aside to test the correction, and shows how far the probabilities were off before and after.

The optional AI Eval Decision Calibration submodule applies a correction to the field automators you select on its Callers tab. It applies only when the question wording, provider, and model match. The answer keeps the model's own probability next to the corrected one. To check that the correction still helps, people review new answers recorded after the correction was built. Once 30 of them are reviewed, with both yes and no answers among them, the Callers tab warns when the correction makes the probabilities worse.

Record decisions can keep the text sent to the model for review, per caller and off by default. The text is kept until its entity is deleted. A caller can also stop recording after a number of answers. Reviewers see the kept text under each answer, a progress line toward the 30 reviews a fit needs, and a Reviewed tab with each label next to the model's probability and the corrected one.

Find out why it fails

Import traces, the recorded conversations between users and your AI feature, review them, and label what happened. Add failure modes to group recurring problems. Promote a conversation into a dataset so the same failure becomes a question in later runs.

On 1.1.x you can undo a trace import. Disabling a failure mode retires it: people can no longer pick it, and labels that already use it keep it. When several people annotate the same question, failure rates count it once, using the earliest annotation.

Trace import on 1.1.x reads the structured JSON that AI Observability writes and shows tool calls on the review card. It reports the parts of a file it could not read instead of dropping them. A trace that was cut short or has tool calls cannot be promoted yet.

Graders and rubrics

Twelve graders ship with the module. Seven LLM judges measure relevance, completeness, accuracy, whether the answer gives the reader useful next steps, fact match against verified facts, whether claims are supported by source material, and a decision rubric. Five deterministic graders check the answer format, the rubric checks described below, which tools the agent called, recall@k, and reciprocal rank.

Rubrics let many questions share the same expectations. The editor provides form fields for the two text-list check kinds and JSON editing for every other kind. The rubric page shows the whole rubric and provides an Edit button.

Improve the prompt

The optimizer reads failed answers and asks a model to propose a better system prompt. Review a proposal before you apply it. The optimizer repeats the current prompt and each candidate because judge scores can vary between runs. It accepts a candidate only when the measured improvement clears the configured margin.

Import and export

Export completed runs as Every Eval Ever (EEE) 0.2.2 JSON files. Import EEE files into the results table to inspect results produced elsewhere and compare results from different test tools. EEE files contain aggregate metrics and run details, not question or answer text. You can also import OpenTelemetry AI traces from a JSON file or directory. On 1.1.x the import page offers a path on the server only to users with Administer AI Eval.

Drush commands

• ai-eval:run (aer): run tests or preview what would run.
• ai-eval:optimize (aeo): generate and measure candidate prompts.
• ai-eval:distill (aed): turn stored results into summary files.
• ai-eval:sample-traces (aest): export answers for human labeling.
• ai-eval:validate-judge (aevj): compare a judge with human labels.
• ai-eval:compare-judges (aecj, 1.1.x): compare two judges on the same labeled answers.
• ai-eval:judges (aej): list judge trust and validation results.
• ai-eval:export-envelope (aee): export stored results as EEE files.
• ai-eval:import-envelope (aeie): import EEE result files.
• ai-eval:import-traces (aeit): import OpenTelemetry AI traces.
• ai-eval:clear-traces (aect, 1.1.x): undo a trace import by its batch id.
• ai-eval:target-fixtures (aetf): list the proving-ground fixtures a test requires.

Requirements

• Drupal 11.3 or later
• PHP 8.3 or later
• AI 1.0 or later

Agent mode also requires AI Agents. Trace review can use AI Observability and OpenTelemetry. Each workflow submodule needs the module it tests: AI Answers, Search API, AI Content Review, or AI Automators (part of AI) with Token. These features are optional.

Which version

Neither branch has a stable release yet, so name the version when you install.

• 1.1.x (alpha) adds workflow mode and decision calibration: composer require 'drupal/ai_eval:^1.1@alpha'
• 1.0.x (beta) has agent mode and chat mode: composer require 'drupal/ai_eval:~1.0.0@beta'

Documentation

The published documentation covers the 1.1.x branch: installation, configuration, usage, and extension guides. For the 1.0.x beta, read the changelog lists what changed in each release.