ai_eval 1.1.0-alpha6

Upgrading from 1.1.0-alpha5

No update hooks, configuration or database schema changes. The Eval Commons schemas change only in a description and their $id (see below).

• ai_eval_droost needs Droost 2.0.0 or later and Tool API. On a site with Droost 1, run composer require -W 'drupal/droost:^2.0' 'drupal/tool:^1.0@beta' (-W because Droost 2.0.0 conflicts with twig 3.30.0), then enable tool, droost_build for View specs and droost_canvas for Canvas specs. Until then the status report shows an error and the grader fails every question with "ai_eval_droost needs Droost 2 and Tool API." View specs grade from Drush only (as uid 1); in a web request they fail with a reason that says so.
• Agent tests now send the question as chat messages, as the AI Assistant does, instead of a wrapped task. Agent scores can change, also for single-turn questions: run agent tests again before you compare them with older runs.
• Images in questions: a question's files now reach the model and the LLM judges when they are images. A files value that is not an image reference is still accepted and not sent. Whether a model can read images comes from the provider's list; some providers list every model.
• Results and envelopes record each image's name, hash, type and size, never its contents. In an every_eval_ever envelope they are in metric_config.additional_details.files.
• Schemas: every schema/*.schema.json $id names 1.1.0-alpha6. dataset.schema.json describes what files does now; no structural change.

Added

• Dataset questions can carry images (png, jpeg, webp, gif). Chat, agent and AI Prompt targets send them to the model, and the LLM judges see them too. (#3594659)

Changed

• ai_eval_droost requires Droost 2 and Tool API; View specs grade from Drush only. (#3594869)

Fixed

• Reviews of text-only questions stay usable for judge validation after the dataset changes or is deleted. (#3594659)
• Results pages limit inline image previews to 256 KB each and 2 MB per page, and show the name and size when a preview is too large. (#3594659)
• Agent tests send the question as chat messages, as the AI Assistant does, instead of a wrapped task. Agent scores can change, also for single-turn questions. (#3594873)
• Fail Canvas assertions when a tree read returns no item list, and fail every View assertion when its configuration read fails. (#3594869)

Documentation

• New tutorial "Test an AI feature that reads images": a chat test with photos, a judge that reads them and a deterministic check of an image safety guardrail, plus a note in "Your first test" that agents get questions as chat messages. (#3594874)
• The install page adds -W to the Droost 2 install and enables droost_build and droost_canvas. The project page describes images in questions and the agent change. The developer docs in docs/semantic/ match this code again.
• The rubric pages list markdown_structure with the check kinds that run, with an example. They no longer point to the closed #3594701 for command and score_delta checks: ai_eval has no sandbox and does not run them, so check site actions with droost_state and code with an outside test runner whose results you import. (#3594872)
• The user docs match the code again: grader counts and the full catalog, judge trust with the refusal setting, the optimizer's in-memory candidates, promotion refusal flags, composer suggestions, the four alter hooks, two missing settings, and resuming a validation. (#3594872)