llm-judging.md: add rule 28 (api-refusal drop class) to Statistics-lens Enforcement riders + matching lens sentence
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Provenance
Surfaced as a prose workflow-fix follow-up by the #2203 Phase-2 Statistics critic (2026-08-09). Not a #2203 experiment defect — a gap in the workflow surface itself.
Gap
.claude/rules/llm-judging.md Enforcement section lists rules 23 / 24 / 26 as Statistics-lens plan riders, but NOT rule 28 (the api-refusal drop class: Batch rows returning stop_reason == "refusal" with empty content, added #2151). A plan whose judged DV scores harmfulness/jailbreak/adversarial-role-play completions can therefore omit api-refusal accounting entirely, and the rule-26 pilot gate PASSes that shape BY DESIGN (rule 28's own non-coverage note). This is a real, outcome-correlated bias: on the #1739 workload 34.1% of harm-judge draws (15,091/44,310) were api-refusal-censored, with the highest-harm rows censored first — biasing the harm rate DOWN precisely on the high-harm arms. It went uncaught by the #2203 v1–v4 mechanical + fact-check passes and only the Statistics critic flagged it; it will recur on every defensive-robustness eval plan.
Fix (scope)
- Add a one-line Statistics-lens Enforcement rider to
.claude/rules/llm-judging.md: a plan whose judged DV scores harm / jailbreak / adversarial-role-play completions names its api-refusal accounting (per-armn_api_refusal) + a sync re-issue remediation at the identical instrument (refscripts/issue1739_evilood_refusal_rejudge.py), or states the exemption. - Add a matching sentence to the Statistics & Measurement lens rubric (
.claude/agents/statistics-critic.mdand/or.claude/agents/critic.mdStatistics lens +.claude/rules/critic-lens-reference.md) so the lens checks it. - OPTIONAL (evaluate cost/benefit): a grep-able
verify_plan.pycheck — a plan whose §6 judged DV text matches harm/jailbreak/adversarial AND has noapi.refusal/stop_reason == refusal/n_api_refusalhandling sentence WARNs.
References
- #2203 (surfaced this gap; Statistics critic Must-Fix 1)
- #2151 (rule 28 origin)
- #1739 (the censoring incident + the
issue1739_evilood_refusal_rejudge.pysync re-issue remediation)
Acceptance
- llm-judging.md Enforcement + the Statistics lens rubric name rule 28's plan-rider obligation.
- If check (3) is implemented: a fixture plan scoring jailbreak completions with no api-refusal handling WARNs; one with the handling sentence passes.