Document how to run evals in various scenarios

Thank you webchick I really enjoy collaborating with you. I am picking this up. I understand that the momentum presses to create the eval runners first and I totally agree. I will need to do some more work here in this issue in parallel with the ones you mention.

Wanted to flag a natural upstream input source for Scenario 3 (catching regressions / new problems).

When an experienced developer corrects an AI agent during a live session — "that's not how cache tags work" or "that hook was replaced in Drupal 11" — that correction is a ready-made eval case waiting to happen. The problem is there's currently no standardized way to capture these corrections so they can feed into evals.

I've opened the issue for capturing expert corrections proposing a capturing-expert-corrections skill that logs corrections as structured JSONL. Each entry includes what was wrong, what's correct, which subsystem, and a failure classification.

The bridge to this issue's Scenario 3 is direct:

1. Expert catches agent error → correction logged with classification
2. Correction log entry becomes the basis for a new eval case
3. Eval confirms the fix actually improves output
4. Scenario 3 documentation can reference correction logs as the "how did we know we needed this eval?" origin story

This gives Scenario 3 a concrete, repeatable workflow instead of relying on someone manually noticing "the agent keeps doing X wrong." The correction log is the early warning system; evals are the regression gate.

Happy to help draft the Scenario 3 section with correction-log-driven examples if that would be useful.

Alex that maps well to what compare.py already supports. A correction log entry becomes an evals.json case with must_contain/must_not_contain checks. Go ahead and draft the Scenario 3 section. I'll wire it into the comparison framework once #3582953: Document how to run evals in various scenarios merges.

If you want to take a look at the correction to eval flow, the CONTRIBUTING.md in the [MR !2](https://git.drupalcode.org/project/ai_best_practices/-/merge_requests/2) has the "Adding a new eval from scratch" walkthrough.

zorz- here you go!

Scenario 3: a live correction becomes a regression test

When an experienced contributor corrects an AI agent during a real Drupal session, treat that correction as the seed of a new behavioral eval. The goal is not just to note "the model got this wrong", but to convert the mistake into a repeatable case that fails before the guidance change and passes after it.

Workflow

1. Capture the correction. Record the original task, the incorrect claim or code, the corrected answer, the subsystem, and a short note explaining why the original reasoning failed.
2. Turn it into an evals.json case. Rephrase the real task as the eval prompt. Use must_contain_any for the required fix, must_not_contain for the bad pattern, and check_php_lint when the response generates PHP.
3. Run the comparison. If this is an existing skill, compare the old version against the edited version with python3 evals/compare.py --skill ... --runs 3. If this is a new skill, compare a no-skill baseline against the skill with --no-baseline.
4. Review the delta. A successful fix should improve the new case without regressing existing cases. Include the comparison output in the merge request so reviewers can see pass-rate delta, token usage, and cost.

What the eval should contain

• Prompt: ask the model to do the same kind of task that triggered the correction
• must_contain_any: the corrected API, cacheability metadata, or other required pattern
• must_not_contain: the stale API, invalid assumption, or unsafe shortcut that was corrected
• check_php_lint: enable when the model outputs PHP code

The "Adding a new eval from scratch" walkthrough in CONTRIBUTING.md already covers the file layout and commands. This Scenario 3 section documents the missing part: how a real correction during live use becomes the next regression test.

This keeps Scenario 3 grounded in real failures from real contributor workflows while reusing the comparison framework that already exists in compare.py. The correction log is the intake layer; evals.json is the regression layer.

Heads up on an isolation issue I found while building the documentation evals for #3583241.

claude -p (pipe mode) loads all enabled plugins, hooks, skills, and MCP servers from ~/.claude/settings.json by default. There is no visual indication this is happening since pipe mode has no UI. That means every eval run with compare.py so far included whatever plugins the person running it had installed as unmeasured context in both configs.

The fix is two flags added to the claude -p invocation:

• --setting-sources "" blocks all user/project/local settings (plugins, hooks, skills, CLAUDE.md, rules)
• --strict-mcp-config blocks all MCP servers

The good news: A/B deltas were always clean because both configs got the same contamination. So comparative conclusions ("skill X adds Y% improvement") still hold. The bad news: absolute pass rates may have been inflated by plugin context that was silently injected.

For reference, when I ran the coding-standards evals in #3583192, I had 13 enabled plugins including one that injects 16 Drupal-specific skills. The Sonnet 24/24 result was real in terms of the delta (0% with or without the coding-standards skill), but the baseline was not a clean "no guidance" baseline.

The fix is included in MR !5 on #3583241. I also added a --cwd flag so the model runs from a neutral directory and cannot see the evals/ or skills/ directories in the repo.