The metrics job on the 1.0.0-alpha16 tag pipeline passed with a green tick while docs/metrics.md was 86 files and 9,283 code lines out of date. Pipeline 968541, job 12311669.

The job cannot fail. Three separate reasons, each sufficient on its own:

  • The comparison ends in || true, so the exit status of the only command that can detect disagreement is discarded.
  • The job carries allow_failure: true, so a broken apt-get, cloc or generator is green too.
  • Nothing in the job asserts anything: it prints a diff and an echo asking a human to read it.

What it printed. The committed page claims 1,198 files and 106,110 lines of code; a re-measurement of the tagged tree gives 1,284 files and 115,393, and the test-to-production ratio moves from 79% to 82%. The page was last regenerated in #3622824 on 2026-09-13, and 35 commits touching 398 files (+27,441/-6,762) have landed since, across three alpha releases, while the snapshot note still reads "Measured on the 1.x branch, 2026-09-13".

Why it was built unable to fail. Two things stop the diff from being an assertion, and the job comment names both: the page counts the file it writes, so a correctly regenerated page does not reproduce byte for byte; and the page does not reproduce across cloc versions, the job installing whatever apt offers (1.98 on this run, against 2.08 on a current workstation). Both are fixable at the source, rather than worked around by making the job unable to fail.

Proposed

  • Exclude docs/metrics.md, the page the generator writes, from its own measurement, so a regeneration of a correct page reproduces that page exactly.
  • Record the cloc version on the page, read from the JSON header cloc already supplies, and have the job install exactly the version the page names. The page then carries what is needed to reproduce it.
  • Drop || true and allow_failure: true: a tag whose metrics page disagrees with the tagged tree is a red pipeline.
  • Run the job on a merge request that touches the page or the generator, as well as on a tag, so the two are checked where they can still be fixed. A merge request touching neither is unaffected, which keeps the once-per-release shape the job was given.
  • Regenerate docs/metrics.md, so the first tag after this change is green on its merits.

AI-Generated: Yes (Claude Code was used to help draft this issue summary and to write the change on the merge request. I reviewed and ran the work myself before posting it.)

Issue fork orchestra-3624570

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mably created an issue. See original summary.

mably’s picture

Status: Active » Needs review

  • mably committed 6e2d2730 on 1.x
    task: #3624570 Let the metrics job fail when the page is stale, and...
mably’s picture

Status: Needs review » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.