Shard i528_phase4_judge.py --backend batch submit (un-sharded single batches.create 409s above the 100k cap); drop its batch-judge lint grandfather
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Problem
scripts/i528_phase4_judge.py (--backend batch) submits its Anthropic Message
Batch un-sharded: a single client.messages.batches.create(...) with all
3 × N requests. On a large i528 job this exceeds the 100k requests/batch API
limit and 409s. It is currently grandfathered in
scripts/workflow_lint.py BATCH_JUDGE_LEGACY_ALLOWLIST.
This is a DISTINCT bug from the #658 wedge (a deadline-less while True poller):
i528_phase4_judge.py --backend batch is submit-only / fire-and-forget — it
creates the batch, writes the batch_id to a pending file, and returns; the
results are merged by a SEPARATE --backend sync re-run (the merge path is not
even in this file). So there is no poller to fix — the latent bug is purely the
un-sharded submit.
Surfaced by
The #658 batch-judge incident cleanup (2026-06-25). Both the workflow-improver
(lint allowlist) and the issue658_judge_e0_batch.py migration implementer
flagged i528_phase4_judge.py as a real pre-#663 judge needing migration, but
deferred it because the submit-then-separate-sync-merge protocol is non-trivial
to change behavior-preservingly.
Fix
Shard the --backend batch submit — either (a) chunk via the shared
_chunk_requests (≤8k/batch) and write multiple batch_ids to the pending file,
or (b) route the submit through the shared #663-hardened client
(src/explore_persona_space/llm/anthropic_client.py AnthropicBatch /
src/explore_persona_space/eval/batch_judge.py). Behavior-preserving constraints
to respect: the custom_id format, the --backend sync merge reader, and the
3×-averaging join. After migration, drop scripts/i528_phase4_judge.py from
BATCH_JUDGE_LEGACY_ALLOWLIST in scripts/workflow_lint.py (the
--check-batch-judge-client guard then enforces it).
Scope / acceptance
- EDIT-ONLY discipline if any i528 batch is ever in flight; do not resubmit a live batch.
- Sharded submit verified by an API-mocked unit test (no live API).
workflow_lint.py --check-batch-judge-clientpasses with the allowlist entry removed.