workflow-fix: watcher auto-terminate+failover for RunPod no-port wedge when poll loop is dead
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by the workflow-fix-on-bug protocol from a workflow-fix candidate raised during the #763 RunPod no-port wedge investigation (user-directed 2026-06-30; subagent diagnosis).
Goal
Promote the watcher wedge arm's confirmed keep_running-False AND inputs_ok case from a reversible _stop_pod to a bounded, idempotent call into the existing backend_poll._failover_wedged_runpod (terminate + fresh re-provision), reusing the durable-lease + sentinel idempotency and per-cell inputs-on-HF gate; every uncertainty path stays ALERT-only.
Workflow gap
- Bug observed: A RunPod pod (#763's z2g3cxyhudmyrf) stuck RUNNING-but-no-port leaked billing and the task sat blocked/alert-only — the poller's terminate+re-provision failover (_failover_wedged_runpod) couldn't run because the per-issue poll loop was dead, and the #692 watcher backstop's strongest action is a gated reversible pod.py stop, never terminate+failover.
- Why it is a workflow gap: The irreversible terminate+re-provision/failover for a no-port wedge lives ONLY in
backend_poll._failover_wedged_runpod, reachable solely while the per-issue poll loop is alive — exactly when a backstop is NOT needed. When the loop is dead (task already blocked //issuepoll chain exited) the #692 watcher backstop can onlystop/alert, and a reversiblestopdoes not heal a host-pinned dead RunPod host (resume returns to the same dead host), so billing leaks until a human terminates (the #763 incident: ~hours of H100 billing on a wedged pod until manual terminate). - Confidence (emitter): medium
Proposed change (candidate diff sketch — refine in planning)
In scripts/autonomous_session_watch.py decide_pod_wedge (:1238) / :3698): add a fourth action _process_wedged_pod (terminate-failover; route the confirmed keep_running is False AND inputs_ok AND matured-wedge case to it (today it returns the reversible ("stop", 0)). The handler calls the EXISTING backend_poll._failover_wedged_runpod(...) (terminate + FRESH re-provision), reusing its durable-lease + sentinel idempotency (_runpod_wedge_already_handled / Lease.runpod_wedge_failover_of) and per-cell inputs-on-HF gate. Bounded at-most-once via the existing sentinel; on no-capacity relaunch it maps to the terminal no_compute_available contract (watcher capacity-retry re-drives). Every uncertainty path (keep_running True/unknown, inputs_ok False) stays ALERT-only.
Scope / surfaces
- Primary:
scripts/autonomous_session_watch.py(decide_pod_wedge,_process_wedged_pod). - Docs:
.claude/rules/compute-backend-failover.md(Part C watcher backstop) + CLAUDE.md "Remaining gap (#667)" note — document that an inputs-safe matured RunPod no-port wedge now auto-terminates+fails-over from the watcher (not just stop/alert); the GCP-#667 frozen-non-terminal-phase gap remains open. - Reuses
backend_poll._failover_wedged_runpod(no new recovery logic — extends WHERE the existing one can fire).
Constraints / invariants
- Workflow-surface only. The terminate-failover action fires ONLY on confirmed
keep_running=False AND inputs_ok=True AND matured wedge; bounded at-most-once (no infinite cascade); every uncertainty path stays ALERT-only (halt-criterion #2 — never strand un-uploaded work). - ARCHITECTURAL ASSESSMENT REQUIRED: this promotes the watcher's auto-action from reversible-only to irreversible terminate+failover. The planner MUST decide if this is a public-contract/architectural change → if so set
architectural: true+ park atplan_pendingfor user greenlight. (Emitter leans IN-SCOPE auto-fix: it reuses the poller's already-irreversible, already-gated, already-idempotent recovery, extending its coverage to when the poll loop is dead — no new contract.) - ruff +
workflow_lint.pypass; add tests for the newdecide_pod_wedgeaction + the bounded-once guard.
Provenance
- workflow_fix_target: scripts/autonomous_session_watch.py, .claude/rules/compute-backend-failover.md, CLAUDE.md
- fingerprint: 3e794ca3af50
Raised by the #763 wedge-investigation subagent (2026-06-30). Related: #667 (GCP sibling gap, still open), #664 + #692 (the existing RunPod wedge poller-failover + watcher-backstop this extends), #763 (incident).