Persist stop_reason per judge draw + truncation-vs-content drop split + rule-26 pilot-gate helper
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Three judge-truncation re-judge waves landed in one week — #1739 (max_tokens=400: 5.4% of 86,521×3 draws truncation-censored, surgical re-judge at 800), #1769 (max_tokens=300 on a multi-field rubric: 1,606/21,000 draws dropped, worst arm 42.5% arm-asymmetric, the whole 21k-call wave re-judged at 600), #1774 (per-draw truncation-recovery merge at 600) — plus two misdiagnosis rounds (#1934/#1773: 2,602 parse failures twice mis-called truncation from log text; a direct Messages-API stop_reason probe showed 39/40 end_turn).
Root cause of the diagnosis cost: the client layer captures stop_reason (src/explore_persona_space/llm/models.py:281 Response.stop_reason; populated at llm/anthropic_client.py:667,1011, llm/openai_client.py:389,659) but every judge module DROPS it before persistence, so drop-class diagnosis (truncation vs content) always needs a fresh API probe or a full re-judge.
- verified-at-filing:
grep -n 'stop_reason' src/explore_persona_space/eval/batch_judge.py src/explore_persona_space/eval/graded_judge.py src/explore_persona_space/eval/judge_dispatch.py src/explore_persona_space/eval/utils.py→ 0 hits in 0 files (absence claim: no judge module persists stop_reason);grep -n 'stop_reason' src/explore_persona_space/llm/models.py src/explore_persona_space/llm/anthropic_client.py→ hits at models.py L281/292/294/312 + anthropic_client.py L667/L1011 (client layer captures it) (2026-08-02).
Goal
Make judge-response truncation mechanically visible in persisted judge results and gateable before a production spend: (1) thread stop_reason into every persisted per-draw judge result, (2) split drop accounting into truncation-drops vs content-drops (alongside the existing content/transport split from #1313), (3) add a judge_pilot_gate helper implementing .claude/rules/llm-judging.md rule 26 (the ≥5k-call pilot gate landed 2026-08-02).
Proposed change (refine in planning)
eval/batch_judge.py/eval/graded_judge.py/eval/judge_dispatch.py: carrystop_reasonfrom the clientResponseinto each stored per-draw result dict — INCLUDING parse-failure records, sostop_reason == "max_tokens"(truncation, budget-class) is mechanically distinguishable fromend_turn(content-class) without a fresh probe.- Drop accounting: add a truncation counter (e.g.
JudgeResult.n_truncation_dropped_draws+ per-item split) alongsiden_dropped_draws(content) andn_transport_lost_draws(#1313). Truncation drops are budget defects (rule 23), not content drops — report all three. - New helper
judge_pilot_gate(...): dispatch N≈100–200 pilot draws spanning the caller's arms at the EXACT production instrument (rubric, judge model, max_tokens) against a pilotcache_dir; return per-arm parse-failure rate + stop_reason tally + a PASS/FAIL verdict (FAIL on any nonzero max_tokens stop_reason, or per-arm parse-fail ≥ ~2% unexplained). Callers use it before any ≥5,000-call production dispatch per rule 26. - Cache hygiene: decide in planning whether truncation-class (
stop_reason == "max_tokens") error dicts should be put-skipped by the rubric-level JudgeCache like transport-class dicts are (#1313), so a budget raise self-heals without a freshcache_dir(today truncation-era parse-error entries are re-served — rule 23's cache caveat).
Constraints / invariants
- Drop-never-coerce (rule 9) and the transport split machinery (#1313) unchanged.
- Rubric-keyed cache KEY semantics (rule 22,
EPM_JUDGE_CACHE_KEY_V2) unchanged — additive result-dict fields only; a key change would cold-invalidate every cache. - New report fields ride rule 18's pinned-report list;
tests/pin the truncation/content/transport split. scripts/workflow_lint.pyno-flags run passes; ruff on touched files passes.