daily-fix: judge max_tokens floor 600 for JSON rubrics
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by /daily 2026-07-30 (problem sweep; miner-3 P1, session d0fe5a10 / issue #1769).
Goal
Raise the llm-judging rule-23 response-budget default for reason-then-score JSON rubrics from ≥ ~300 to ≥ ~600 tokens, and thread the higher default through the judge call sites that use such rubrics.
Workflow gap
- Bug observed: the #1769 fu1 judge run (21,000 calls, reason-then-score JSON rubric) ran at the rule-23 floor
max_tokens=300and truncation-censored 12.5% of hallucination draws (874/7000 parse errors, arm-asymmetric up to 42.5% onhallucination/decode_only/a3), forcing a full mt600 re-judge (~3h wall + a second batch spend). Second measured floor failure: #1739's wave at 400 tokens censored 5.4% of draws the same week. - Why it is a workflow gap: rule 23's "≥ ~300" floor is now measurably insufficient for the multi-field JSON reason-then-score rubrics the project actually runs; each failure costs a full re-judge batch and risks the selection-artifact-mimicking censoring rule 23 exists to prevent.
- Confidence (emitter): medium
- verified-at-filing:
grep -n '300 response tokens' .claude/rules/llm-judging.md→ line 197 (**≥ ~300 response tokens** (the #1090 recovery) + the enforcement echo at line 457 — the 300 floor is still the stated default (2026-07-31 filing time). The #1769 drop numbers are quoted from that session's own drop-stat tool results (unverified hypothesis — verify at plan time: exact per-arm rates fromeval_results/issue_1769/fu1 judge drop stats).
Proposed change (candidate diff sketch — refine in planning)
Rule 23: state ≥ ~600 as the default for reason-then-score JSON (multi-field) rubrics, keeping ≥ ~300 for short single-rationale rubrics and keeping the post-run per-arm drop re-measure as the binding check; cite the two measured failures (#1739 at 400, #1769 at 300). Optionally raise the graded_judge.judge_graded recommended caller value in the same pass.
Scope / surfaces
- Primary target:
.claude/rules/llm-judging.md(rule 23 + its enforcement bullet) - Secondary: the rule-23 cross-references in CLAUDE.md § Measurement validity and
experiment-guidelines.mditem 9 if they quote the floor (grep~300across the workflow surface and update every hit).
Constraints / invariants
- The floor stays a floor; the post-resize per-arm drop re-measure stays the binding check (rule 23's #1739 clause).
- Cache caveat unchanged (rubric-level cache does not key on max_tokens — re-judge guidance stays).
Provenance
-
fingerprint: 85e9a904e86c
-
workflow_fix_target: .claude/rules/llm-judging.md
-
origin: /daily 2026-07-30 miner-3 P1 (transcript d0fe5a10, issue #1769 fu1 judge round)