Batch judge dispatch sends temperature to opus-4-8 (rejected) — opus unusable as batch judge/kappa-control
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Problem
The shared batch-judge dispatch path attaches temperature (the judge instrument's temp 0) to every request. claude-opus-4-8 rejects the temperature parameter entirely — the Batch API returns, per request:
InvalidRequestError(message='`temperature` is deprecated for this model.', type='invalid_request_error')
Result: every opus-4-8 batch judge request errors (errored=N, succeeded=0), the rows are quarantined (invalid_request_error, NOT retried), and any opus-4-8 κ-calibration control silently produces n_paired=0 / cohen_kappa_em=null.
Reproduction (verified on 2026-08-20, task #2379)
- opus-4-8 SYNC call, no
temperature: works (stop_reason=end_turn). - opus-4-8 SYNC call, no
temperature, WITHcache_controlephemeral on system: works. - opus-4-8 BATCH via the judge dispatch (sends
temperature=0): all requestsinvalid_request_error(temperature is deprecated for this model). - claude-sonnet-4-5-20250929 BATCH via the same path: works (temperature accepted).
Evidence in eval_results/issue_2379/: kappa_control.json (n_paired=0, cohen_kappa_em=null, status:"OK" — see bug 2 below), kappa/opus_align/save_raw.json + kappa/opus_coh/save_raw.json (all error:true, reasoning:"batch_error: invalid_request_error (quarantined)"), kappa/opus_*/ckpt/*/quarantine.json (2000 custom_ids each). Failed batches msgbatch_01NGJh2azE7C6v94aQ35gjnz (opus_align), msgbatch_01UH88KjPyL2FxcjdX9kG4zZ (opus_coh).
Impact
Fleet-wide: opus-4-8 (and any temperature-deprecating model, e.g. the opus-5 / newer families) cannot be used as a batch judge or κ-calibration control through the shared dispatch. Every task that pins opus for a κ-control against the sonnet primary judge hits this. #2379's κ-control is carried as a caveat because of it.
Fix direction
- In the judge/dispatch request builder (locate: the layer that sets
temperatureon the judge Message request — batch_judge deliberately excludes it from its cache key, so it is attached in the judge instrument caller orllm/api_dispatch.py), OMITtemperaturefor models that deprecate it. Prefer a capability check / model-family allowlist over a hardcoded id so opus-5 / future families are covered. Confirm sonnet-4-5 and other current judges still receivetemperature(they require it for deterministic temp-0 judging). - The κ phase (
scripts/issue2379_judge.py::phase_kappa) writesstatus:"OK"even when EVERY opus request errored andn_paired=0— it should writestatus:"failed"(ordegraded) with the error class, so a downstream reader is not misled bystatus:"OK"beside a null κ. Consider lifting this into the shared judge helper so all κ-controls fail loud.
Acceptance
- An opus-4-8 batch judge dispatch succeeds (a small live smoke: N≈4 opus-4-8 batch requests return
succeeded, noterrored). - sonnet-4-5 batch judging is unchanged (regression check).
- The κ phase writes a non-OK status when the control judge produces zero paired rows.