poll_pipeline ETA tripwire self-disables on plan §9 calendar-latency rows; range cells silently parse as lower bound
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Goal
Make poll_pipeline.py's phase-ETA tripwire survive plan §9 compute tables that legitimately contain non-numeric latency cells, and stop a range cell from being silently parsed as its lower bound. Today the tripwire disables itself fleet-wide on (probably) every plan that uses the Anthropic Batch API for judging, which is most kind: experiment plans.
Evidence (observed live on #2162, 2026-08-07)
poll_pipeline.py emitted, on every tick of a healthy 8x H100 run:
INFO poll_pipeline: no parseable §9 planned_wall_h for #2162; phase-ETA tripwire disabled (fail-safe)
The plan is NOT missing the column — tasks/running/2162/plans/plan.md line 351 is
| component | planned_wall_h | planned_gpu_h | parallelism | basis |.
Running the parser directly against the plan text isolates the offending row:
_md_planned_wall_rows(plan) RAISED _UnparseableWallRow:
| P9 stage-2 judge (≈36.3k calls) | ≤24 calendar | 0 | Batch API | same instrument |
The planned_wall_h cell is ≤24 calendar. _LEADING_FLOAT_RE does not match a leading ≤, so
_md_planned_wall_rows raises _UnparseableWallRow, and the caller maps that to None —
disabling the whole tripwire, by deliberate design (the docstring at scripts/poll_pipeline.py
~L3872-3881 says it raises rather than return a partial sum, "AC #2").
Two distinct defects
-
Hard self-disable. ONE non-numeric cell anywhere in the table turns off ETA-deviation detection for the ENTIRE run. The fail-safe is defensible in isolation (never a partial sum), but the trigger is a LEGITIMATE plan authoring choice, not a malformed plan: a Batch-API judge phase's cost is calendar latency (SLA 2-24 h), not GPU wall-hours, and writing a number there would be false.
-
Silent lower-bound mis-parse — worse than the disable. The sibling row in the same table,
| P6 judge waves (pilot + gate-3 sync + batch) | 2–24 calendar | 0 | ... |, has a cell beginning with the digit2, so_LEADING_FLOAT_REDOES match and it parses as2.0— contributing a bogus 2 h to the planned total and silently discarding the–24 calendarremainder. Had the P9 row been written2–24 calendarinstead of≤24 calendar, the tripwire would have stayed ENABLED with a wrong (understated) budget, which fires spurious ETA-deviation posts instead of disabling. So the current behavior is bimodal and both branches are wrong: a leading digit gives a silently wrong budget; a leading≤gives no budget at all.
Why this is systematic, not #2162-specific
Every kind: experiment plan that routes judging through the Batch API (the standing CLAUDE.md
mandate for large judge sets) has at least one phase whose honest wall figure is a calendar-latency
range or bound. The planner has no documented way to express "this phase costs no GPU wall-time,
it costs up to 24 h of calendar latency" in a planned_wall_h cell, so authors write prose. The
poller then either disables or mis-sums. Worth checking how many recent plans' tripwires are
silently off.
Proposed fix (direction, not a mandate — the spawned session's planner + critics decide)
- Poller side: recognize a documented set of non-wall sentinels in a
planned_wall_hcell (e.g. a cell with no leading float that matches acalendar/n/amarker) and SKIP that row rather than disabling the tripwire, while still hard-raising_UnparseableWallRowon a genuinely malformed numeric cell (preserving the AC #2 intent: never a silent partial sum from a MALFORMED row, but do not conflate "deliberately not a wall figure" with "malformed"). - Reject ambiguity rather than guessing: a range cell (
2–24,2-24) must NOT parse to its lower bound. Either raise, or require an explicit single figure. The current silent lower-bound behavior is the more dangerous of the two defects. - Planner side: give §9 an explicit convention for calendar-latency phases — e.g. a separate
calendar_latency_hcolumn, or a required literal in theplanned_wall_hcell — and state it in.claude/rules/plan-compute-sizing.mdso the poller contract is authored against rather than discovered. Whatever literal is chosen must be pinned by a test on BOTH sides so the two surfaces cannot drift. - Add regression fixtures for: a
≤24 calendarcell, a2–24 calendarcell, a genuinely malformed cell (must still raise), and an all-numeric table (must still sum).
Impact / severity
Monitoring-only; no data or spend at risk directly. Stall detection (pid liveness + log mtime) is unaffected and still fired correctly throughout #2162. The loss is the ETA-deviation auto-post, which is the mechanism that would catch a throughput blowout on a multi-hour run. #2162 retains independent protection (its plan's own throughput pilot gate re-derives the fence at 2x the pilot-extrapolated wall), so this is a degraded-defence-in-depth finding, not an active incident.
Provenance
Surfaced by the #2162 orchestrator while the run was healthy in P2. Parser located at
scripts/poll_pipeline.py _md_planned_wall_rows (~L3872-3906), caller _parse_plan_wall_budget /
_plan_total_wall_h_for_issue; both returned None for #2162's plan. _html_planned_wall_rows
returned [] (the plan's table is markdown).