ai_eval 1.1.0-alpha5

Upgrading from 1.1.0-alpha4

No update hooks, configuration or database schema changes, and no change to the Eval Commons schemas. Some screens read the same data differently:

• Gold labels and judge validation count only final answers. Labels on tool steps (an output that is only "assistant:", or with tool calls) and on embeddings calls stay in the database but no longer count. A judge's gold coverage can drop; validate it again if you rely on it.
• Promotion refuses an embeddings call and a tool step whose call was not recorded, and says why on the trace card. Questions promoted earlier stay as they are. Promote all on an agent run takes its final answer, with the visitor's first message as the input.
• Annotate progress counts the answers of the latest run of a test, so a test that showed Done can show In progress after a new run. The step's progress array no longer has suggested.
• Failure rates split the old "(untagged)" row into "Passed", "Failed, no failure mode" and "Deferred". Each rate row has a new verdict key.
• Results show the gate a run recorded when it ran. An older result can now show a different threshold than before, because it used to show the test's current gate.

Added

• Warn on trace cards when captured tool turns prevent promotion, with the cut notice taking precedence. (#3594850)
• Show the visitor's first question on trace review group cards, shortened to 120 characters. (#3594839)

Fixed

• Skip symlinked envelope folders during uninstall, preserve file names with trailing newlines, and escape closing prompt fences with whitespace without changing unrelated tags. (#3594853)
• Serialize promotions per destination and count busy bulk items as refused while continuing the run. (#3594841)
• Keep agent calls in insertion order when timestamps tie, refuse missing or cut first questions, and retain the first input's provenance. (#3594838)
• Replace completed promotion controls with a dataset confirmation and question link, and serialize duplicate checks and inserts. (#3594841)
• Promote an agent run's final answer with its first user message and preserve the group's failure mode. (#3594838)
• Show imported embeddings as having no text output and keep producer diagnostics out of promoted answers. (#3594840)
• Resume unfinished judge validation on the judge page, hide the launch form while it is open, and offer Cancel. (#3594845)
• Use only non-empty replies without tool calls for gold labeling, coverage, agreement, and judge validation; report excluded labels without deleting them. (#3594842)
• Advance past labeled and deferred answers on the gold screen, with a deferred count and a review link at completion. (#3594844)
• Keep gold and re-label navigation stable after a verdict, advance on save, and offer Next unlabeled. (#3594843)
• Use singular counts and human entity labels in admin text. Clarify the configurable trace limit. (#3594852)
• Explain the manual model fallback when no chat models are listed. (#3594851)
• Avoid a contradictory stopped-at count above the recording limit. (#3594849)
• Keep gate threshold defaults on the selected metric scale. (#3594848)
• Add rubric phrase rows without saving and prevent Enter from saving. (#3594847)
• Give question row actions accessible names using the question id. (#3594846)
• Show the gate a run used, from its recorded settings, or "Gate not recorded" instead of an invented 4.0. (#3594855)
• Say that hard and soft gates only change the exit code of drush ai-eval:run. (#3594856)
• Count Annotate progress against the answers of the latest run; deferred answers count as labeled. (#3594857)
• Split failure-rate rows without a failure mode into "Passed", "Failed, no failure mode" and "Deferred"; the shares still add up to 100%. (#3594858)
• Say when saving a calibration replaced an earlier fit that was not active. (#3594859)
• Warn that the judge does not apply a fit on earlier rubric wording. (#3594860)
• Show test, judge and gate metric names on the results screens instead of their ids. (#3594861)
• Use plain labels for the differences on the compare page; raw values move to a closed "Technical details". (#3594862)
• Stop counting legacy tool steps, stored as "assistant:" alone, as final answers in gold labeling, agreement and judge validation. Refuse to promote such a step or an embeddings call, and say why on the trace card; Promote all picks the same final answer. On the compare page show a value once when both runs read the same. (#3594867)

Documentation

• Eight step-by-step tutorials walk one site from a first test to a calibrated decision judge: Your first test; Improve your test; Read a result and record your verdict; From real traffic to tests; Check a judge against your own labels; More ways to trust a judge; Editorial rules as a judge; Calibrate a decision judge. Each explains the mechanics in collapsible panels and ends with self-check questions. (#3594854)
• A new Glossary page defines the words the screens use. Tutorial terms link to it and show the short definition as a tooltip. (#3594854)
• The project page describes what this release changes: final answers only for gold labels, promotion refusals, and the new tutorials. The developer docs in docs/semantic/ match this code again.