workflow-fix: verify_plan WARN check for sub-resolution judge-pilot sizing (per-arm draws below floor(1/threshold)+1)
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Goal
Add a WARN-only verify_plan.py check that flags a plan whose judge-pilot gate is UNSATISFIABLE BY ARITHMETIC — the per-arm effective draw count sits below the resolution its own pass/fail threshold requires — so this class is caught at plan time instead of by a reviewer lens (or, worse, by a false HALT mid-round).
The defect class
.claude/rules/llm-judging.md rule 26 (#2021, sizing clause #2124) requires every ≳5,000-call judge wave to be pilot-gated before the production dispatch. A plan hand-specifies the pilot's draw budget and its per-arm gate threshold independently. When
per_arm_effective_draws < floor(1 / per_arm_threshold) + 1
the gate cannot work in either direction:
- ONE parse failure already exceeds the threshold, so the gate FAILs as a pure granularity artifact and freezes a production wave that is fine; and
- a zero-failure PASS carries no evidence the true rate is below threshold — the observed rate simply has no resolution at that sample size.
Either way the gate is decorative-or-harmful rather than protective, which is the same structural defect as a null-band-free selection read: a check whose arithmetic cannot support the verdict it issues.
Founding + recurrence
- #2124 established the sizing clause.
- #2162 plan v7 (
turn-boundary-multipatchfollow-up round, post-approval panel 2026-08-14) reproduced it POST-#2124: §7.3 gatedparse-fail < 2% per armon 150 total draws spanning 3 arms ⇒ ~30 value-rubric draws per arm, against the 51 required at a 2% threshold (floor(1/0.02)+1). Caught by thestatistics-criticas must-fix 4 and confirmed by the orchestrator's own arithmetic; the plan's wave-2 was 21,060 calls, so a granularity-artifact HALT would have frozen the entire depth read.
The recurrence after the rule already existed is the argument for a mechanical check: the rule tells an author to pilot-gate, but nothing checks that the two numbers they choose are mutually satisfiable.
Fix
Add a WARN-only check to scripts/verify_plan.py (next free check id, following the existing c43/c46/c50 WARN-only precedents):
- Parse the plan for a judge-pilot gate specification — the draw budget, the per-arm threshold, and the arm/rubric count over which the budget is split. The #2162 v7 shape (§7.3 "150 draws … parse-fail < 2% per arm", with the arm count derivable from the same section's rubric enumeration) is the reference fixture; expect prose variation and prefer a conservative parse that stays silent when it cannot confidently extract all three numbers.
- WARN when
per_arm_effective_draws < floor(1/threshold) + 1, naming all three parsed numbers, the computed floor, and the suggested budget. - Point the message at
eval.judge_pilot.judge_pilot_gate, which already refuses unsatisfiable configs at config time — the check should route authors toward the existing helper rather than toward a hand-recomputed budget.
WARN-only is deliberate and load-bearing. verify_plan.py's no-flags run feeds the Step 9c gate; a FAIL-posture check with a prose-parsing front end is the #1388 fleet-wedge shape. The binding gate stays with the statistics-critic lens; this check exists to catch the class earlier and cheaply.
Pin with a test carrying the #2162 v7 sizing as a fixture (150 draws / 3 arms / 2% ⇒ WARN) plus a satisfiable counter-fixture (≥51 per arm ⇒ silent) and an unparseable-plan fixture (⇒ silent, no false WARN).
Provenance
Surfaced as an explicit workflow-surface prose follow-up by the statistics-critic during #2162's turn-boundary-multipatch post-approval critic panel, 2026-08-14, under its own verdict's "Workflow-surface prose follow-up (per workflow-fix-on-bug; orchestrator routes, not me)" heading — correct: subagents never file, the orchestrator routes (CLAUDE.md § Workflow-fix-on-bug protocol, surfaced-prose clause). The underlying must-fix was independently confirmed by the orchestrator before filing: plan v7 line 481-484 specifies the 150-draw / 2%-per-arm gate, and 150 draws over 3 arms cannot resolve a 2% threshold.
Dedup: distinct from #2298 (filed the same session — stale fellows-first auto-lane prose in critic-lens-reference.md); different target file, different defect class.
workflow_fix_target: scripts/verify_plan.py