workflow-fix: zombie-GPU stalled override needs a material-CPU-delta veto
kind: infra#wf-fix#wf-fix-fp:994d4d74543e
track:Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by the workflow-fix-on-bug protocol from an orchestrator observation on task #825 (run-1e fit phase, 2026-07-03).
Goal
Harden poll_pipeline.py's zombie-GPU stalled-override predicate: do not override running->stalled when the workload demonstrably computes.
Workflow gap
- Bug observed: #825 run-1e (fit phase): the 2-tick zombie-GPU override (#826) fired running->stalled on a 1816 MiB host-namespace VRAM leftover from the PRIOR dead run while the LIVE fit process burned 186% CPU and wrote 5 per-cell result JSONs within 15 min. session_cpu_secs advanced +1102s/+989s across the two ticks the persistence counted.
- Why it is a workflow gap: the override deliberately distrusts cpu_advancing because #664's hung generate() churned ~22% CPU — but a MATERIAL CPU delta (>= ~0.5 cores sustained across both ticks) is real compute, not churn, and a stale VRAM holder from a previous crashed run on the same pod is a common state after a crash-fix relaunch. Each false stalled tick invites failure routing against a healthy run.
- Confidence (emitter): medium
Proposed change (candidate diff sketch — refine in planning)
In the 2-tick zombie-GPU override (poll_pipeline.py, #826 block):
+ cpu_delta_cores = (cpu_now - cpu_prev) / tick_seconds
+ if cpu_delta_cores >= EPM_ZOMBIE_OVERRIDE_CPU_CORES_MIN (default 0.5)
+ on BOTH persisted ticks: liveness veto (log, do not override).
Optionally: when the flagged VRAM PID predates the CURRENT run's
epm:run-launched ts (prior-run leftover), require a higher persistence
count or exempt it from the override.
Scope / surfaces
- Primary target:
scripts/poll_pipeline.py(the #826 zombie-GPU override block + its liveness-veto helper). - Tests:
tests/poller unit coverage for the override predicate. - Grep:
grep -n "zombie" scripts/poll_pipeline.py.
Constraints / invariants
- Must NOT regress the #664 protection: a hung generate() at ~0.22 cores must still override to stalled. The CPU-cores threshold separates the regimes (~0.22 vs ~2.0 measured).
- Workflow-surface only; ruff + workflow lint pass.
Provenance
- workflow_fix_target: scripts/poll_pipeline.py
- fingerprint: 994d4d74543e
Origin: epm:progress v94 FALSE-STALL override note on task #825 (2026-07-03T23:24Z).
Activity