EPS Dashboard

Goal

Make scripts/poll_pipeline.py's state cache correct for the multi-pod (per-arm) case: two or more pollers watching DIFFERENT pods of the SAME issue must not share or overwrite each other's state. Today they silently do, and every phase-transition marker, GPU-idle advisory, ETA-deviation post, and stall verdict downstream of that state is untrustworthy on any multi-pod issue.

The defect

scripts/poll_pipeline.py keys its state cache on the issue number only:

def _load_state(state_file: Path, issue: int) -> dict[str, str]:
    ...
    return data.get(str(issue), {})

def _save_state(state_file: Path, issue: int, payload: dict[str, str]) -> None:
    ...
    all_state[str(issue)] = payload

The default path is <main-checkout>/.claude/cache/poll-pipeline-<N>.json — also per-issue, with no pod dimension. But the poller's UNIT OF WORK is a POD: the CLI requires --pod / --log / --pid-file, and pod.py names a second pod for the same task pod-<N>-<slug>. So an issue running N pods concurrently has N pollers, all defaulting to ONE state file and ONE key.

_save_state ASSIGNS the whole payload (all_state[str(issue)] = payload) rather than merging, so this is not a partial interleave — each tick's write discards the other pod's entire state.

Observed live (#2546, 2026-08-26)

Task #2546 runs three concurrent arms, one 4xH100 pod each: pod-2546-arm1c, pod-2546-arm2, pod-2546-arm3. The default state file held, mid-run:

{ "2546": { "phase": "p2_gen_post_rig", "last_mtime_epoch": "1787721939",
            "gpu_idle_since_epoch": "0", "gpu_idle_escalated_phases": "",
            "eta_budget_warned": "0", "max_cpu_secs": "1992.0", ... } }

p2_gen_post_rig is arm-1's phase. Had the arm-2 poller used the default, its FIRST tick would have read arm-1's phase as its own previous phase — arm 2 was at p1_smoke, so the poller would have posted a spurious p2_gen_post_rig -> p1_smoke transition (a BACKWARD phase move that no dispatcher can produce), then clobbered arm-1's state on save. The orchestrator caught this before arming the second chain and passed an explicit distinct --state-file; the bug is that nothing in the tool or the docs requires that.

Affected state (not only `phase`)

Every key in the payload is shared, so the blast radius is wider than milestone detection:

phase / last_phase_change_epoch — spurious and BACKWARD phase-transition epm:progress markers.
last_mtime_epoch, max_cpu_secs — feed the stall conjunction and the CPU-advancing override; cross-pod values can mask a genuinely stalled pod (a busy sibling's advancing CPU reads as this pod's liveness) or false-flag a healthy one.
gpu_idle_since_epoch, gpu_idle_advised_phases, gpu_idle_escalated_phases, gpu_idle_escalation_counts — the #664 idle-burn escalation channel. A sibling's busy GPUs can reset an idle pod's timer, suppressing the escalation that exists to catch a multi-GPU pod billing at ~0% (the exact #1739 pod-1739-r5ladder failure mode: ~3 h idle at $18.36/hr).
gpu_width_*, gpu_underparallel_warned, eta_budget_warned, eta_deviation_posted_keys, post_done_advisory_posted — per-run dedup flags; one pod's post suppresses the other's.

The idle-escalation suppression is the most consequential: it is a money-leak detector, and cross-contamination fails it OPEN.

Fix (proposed; implementer to confirm shape)

Give the state a POD dimension. Preferred: key the inner dict on (issue, pod) — e.g. f"{issue}:{pod}" — so the default path keeps working and multi-pod becomes correct with NO caller change and no CLI change. Alternative: fold the pod into the default FILENAME (poll-pipeline-<N>-<pod>.json). Either way the explicit --state-file override stays.

Also consider a cheap guard for the residual: record the owning pod inside the payload and, when a load finds a payload stamped with a DIFFERENT pod, treat it as empty and log a warning rather than inheriting it — that makes any future keying regression loud instead of silent.

Migration note: re-keying orphans existing per-issue state. Cost is bounded and one-time — a live poller sees empty state and re-posts one phase transition; no data is lost. Prefer that to a silent read of a foreign pod's state. Do NOT attempt to migrate the old key into a pod-specific one: the old payload has no pod attribution, so which pod it belongs to is unknowable.

Scope

scripts/poll_pipeline.py — _load_state / _save_state / default-path derivation.
A regression test: two pollers, same issue, different pods, sharing one state file — assert neither reads nor overwrites the other's phase and idle-escalation keys (drive the state helpers directly; no SSH needed).
.claude/skills/issue/SKILL.md Step 6d.2 (the bg-Bash poll invocation) — note the multi-pod case so an orchestrator arming a second chain does the right thing even on an unfixed poller.

Backstop already in place today: multi-pod issues are not the common shape, and the single-pod default is correct, so nothing existing regresses.

Provenance

Found by the /issue 2546 orchestrator while arming a second poll chain for arm-2 after arm-1's P1 gates PASSed and arms 2/3 were provisioned (plan section 9 serialization). The three-pod shape is plan-registered for #2546, so this is a designed configuration, not an exotic one.