ai_eval 1.1.0-alpha2

Second alpha of the 1.1.x branch. It adds one optional submodule, AI Eval Decision Calibration (ai_eval_decision_calibration, #3594780), and installation docs that name the version for each branch.

New: decision calibration for selected callers

• Record a sample of the true/false answers a Decision field automator gets from its model. A record keeps the question wording and the entity reference, not the text sent to the model.
• Review the recorded answers without seeing the model's probability.
• Fit a calibration from the reviews. AI Eval holds back part of the answers and shows how far the model's probabilities were off before and after the correction.
• The correction applies only to the callers you select on its Callers tab, and only when the question wording, provider and model that answered match. The answer keeps its raw probability and the calibration ID.
• The Callers tab checks the correction on answers recorded after the fit and then reviewed, and warns when it makes the Brier score worse.

Upgrading from 1.1.0-alpha1

No database updates. The submodule is opt-in; enabling it creates the table for recorded answers.

Developers: CalibrationStoreInterface::save() takes a new optional $thirdPartySettings argument. Custom implementations of the interface must add it. A new hook, hook_ai_eval_calibration_view_alter(), lets a module describe how it uses a calibration on the calibration page.

Upgrading from 1.0.0-beta6

Read the 1.1.0-alpha1 notes as well. In short:

• Run drush updb. All new submodules are opt-in.
• Calibrations now belong to a provider and a model. Calibrations fitted on 1.0.x no longer apply: fit them again.
• Reviews on results stored before AI Eval recorded which judges scored each answer no longer count for any judge's validation. Run the test again and review the new results.
• Developers: several APIs changed, including new required interface methods. See the API notes in the CHANGELOG.

Known limitation

Recorded decision answers do not keep the language of the text, so answers from translations are reviewed and fitted together.

Full list: CHANGELOG.md.