verify_report.py: blob-identity check for body-linked captions.json pins (sibling of image-pin-blob-identity)
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Goal
Catch the case where a report body cites a SHA-pinned figures/**/captions.json
whose content at that pin is superseded — the text-artifact analogue of the
existing image-pin-blob-identity check, which already covers pinned PNGs.
The failure shape (realized in #2329 q35_ladder_decay)
A report round fixed three caption defects, then re-verified clean. But the
body's decay_raw block still cited captions.json at the PRE-FIX revision:
- pinned revision
fa9b14ee16— the production-time commit - the specific claim the citation supports ("the pinned captions.json states they are excluded") is TRUE at that pin, so no accuracy check fires
- yet the same pinned file still carried the three superseded texts the round
existed to scrub:
"pooled per-segment arm means"(the wrong estimator),"n = 21 steered context-end cells"(the wrong population), and"Ten declared panels"(the wrong count)
So a reader following the citation reads exactly the text the report corrected.
Nothing mechanical objected: image-pin-blob-identity covers PNGs only, and the
citation's own claim was accurate at its pin. It was caught by a human verifier
diffing the pin against the worktree tip.
Pinning the production-time declaration is defensible provenance — the fix is not to ban stale pins outright, but to make the divergence VISIBLE so the author chooses knowingly.
Proposed check
For every SHA-pinned link in the report body (and the detailed companion)
resolving to a figures/**/captions.json:
- Read the blob at the pinned SHA and compare against the figures-root copy.
- On a mismatch, report which figure ids'
caption_bulletsdiffer — not just "differs", since a caption file legitimately accumulates entries. - Posture: WARN by default, naming the pin and the diverging ids, and escalate to FAIL only when the pinned revision contains text the CURRENT revision has removed (the superseded-text case above) — that is the shape that actively misleads a reader. A pin that merely lacks LATER-ADDED entries is benign provenance.
Keep it evidence-based and symmetric with the image check: key off the links actually present, so a report citing no captions.json is unaffected.
Acceptance criteria
- Reproduce the #2329 shape: a body citing
captions.jsonat a revision whose entries contain text absent from the figures-root copy ⇒ finding naming the pin and the diverging figure ids. - A body citing
captions.jsonat a revision byte-identical to the figures-root copy ⇒ PASS, silent. - A pin that is merely OLDER but whose overlapping entries agree (only later-added ids missing) ⇒ WARN at most, never FAIL.
- A body citing no
captions.json⇒ no finding (no false positive on absence). - An unresolvable pin (blob absent at that SHA) is reported as unresolvable,
distinctly from a content mismatch — a broken citation is not a stale one
(the existing
stale-evidence-pinscheck already draws this distinction; match its vocabulary). - Tests failing before / passing after; no new red in the no-flags
workflow_lint.pyrun or the mapped-test selection.
Provenance
Surfaced as a prose mechanizable: yes recommendation by report-verifier
during #2329 round q35_ladder_decay report verification (round 2, PASS with one
minor, 2026-08-21). The orchestrator verified both halves before filing — the
claim holds at the old pin, and the three superseded texts are present there and
absent at the corrected revision — then re-pinned the two citations by hand
(45d41b5dad). Evidence: #2329 events.jsonl epm:report-verified round 2;
git diff fa9b14ee16:figures/issue_2329/q35_ladder_decay/captions.json vs the
branch tip on issue-2329-q35-ladder-decay.
- target_file: scripts/verify_report.py
- fingerprint: captions-json-pin-blob-identity-and-superseded-text
- confidence: high — observed on a committed body that had already PASSed every other mechanical check