Problem/Motivation

The --format=llm envelope is the agent-facing surface, and an agent has only the bytes in the report — no memory of what command produced it. Reviewing the current envelope from that consumer's seat found three defects:

  1. A category-filtered run is indistinguishable from a full run. site-doctor:check entropy --format=llm emits an envelope with no scope marker: same keys, exit_code: 0, empty checks_disabled. An agent reads partial coverage as clean full coverage — the same "absence of a finding must never read as a pass" rule the envelope already honors for disabled checks, unhandled for filtered runs.
  2. _meta.severity_semantics misleads on both ends: ok: "clean pass" presents inventory facts (a runtime override exists, a workflow is declared) as endorsements, and info: "advisory, no action required" tells an agent to ignore exactly the findings that carry remediation links.
  3. _meta.exit_codes is a positional array — the code-to-meaning mapping is implicit in array index, a cheap misindexing hazard.

Steps to reproduce

  1. Run drush site-doctor:check --format=llm --no-store and drush site-doctor:check entropy --format=llm --no-store.
  2. Diff the two envelopes: nothing in the second says it covered one category.

Proposed resolution

  • Top-level scope field: "full", or the requested category on filtered runs — plus a _meta.scope line stating that a scoped report says nothing about checks outside it. Mirrors the stored run's complete/partial distinction.
  • _meta.severity_semantics rewording: ok = "inventory fact — records what exists; neither a problem nor an endorsement"; info = "advisory — worth attention, never gates".
  • _meta.exit_codes becomes an object keyed by the literal exit code.

All three are additions or wording changes inside the llm format only; schema_version bumps per the alpha stability policy if reviewers judge the scope key addition to warrant it. The envelope conventions established here carry into the stored-history read commands (#3608643: Add read commands for stored results and changes — see the consumer-review comment there for the full findings list, including design inputs that only apply to that future envelope).

Remaining tasks

  • Implement + kernel tests (scope field on filtered vs full runs; _meta wording assertions).
  • Decide whether scope warrants a schema_version bump (proposed: yes, additive but meaning-bearing).

User interface changes

None — llm format output only.

API changes

New scope key and reworded _meta entries in the llm envelope.

Data model changes

None.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

amangrover90 created an issue. See original summary.

amangrover90’s picture

Title: llm format: declare run scope and fix misleading _meta wording » llm/json format: declare run scope and fix misleading _meta wording

  • amangrover90 committed 08c2ae28 on 1.x
    Resolves #3608884: Declare run scope and fix misleading _meta wording
    
amangrover90’s picture

Status: Active » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.