daily-fix: pod HOLD gates + launches must arm a re-drive
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Workflow gap
Two #1947 pods each burned ~10 h idle on 2026-08-04 because a pod-side state had no VM-side owner:
- pod-1947-r3 (8×H200, ~$32/h): finished smoke+pilot and stopped at its designed circuit-breaker "[launcher] pilot leg rc=0 — HOLD for cost review" at 03:52:43Z — but no review loop was armed; the session's own admission: "I built that gate and never closed the loop on it." Discovered only by Thomas's "check progress" at ~13:58Z.
- pod-1947-loc (4×H200): bootstrapped and never ran any workload — no logs dir, no processes, nothing written in 6 h; the dispatch that should have launched its workload never fired. Terminated on Thomas's "Kill loc immediately — it's pure waste."
verified-at-filing: both are the recovery miner's probed live-API/ssh reads at discovery time (session 024a96de rows 1544–1630: uptime 10.48 h, 0% on 8 GPUs, empty proc list). The pods are long gone; the gap is the pattern.
Proposed change
Two clauses in .claude/rules/pod-side-reporting.md (the (re)launch contract):
- Any pod-side HOLD/gate emission must arm a VM-side re-drive at the moment the HOLD is
written (an
epm:progresshold note naming the reviewer + a Monitor/cron wakeup); a HOLD sentinel with no armed reviewer is itself watcher-escalation material. - Post-provision launch confirmation: the launcher verifies (and records in the
epm:run-launchednote) a live workload pid + first log line within N minutes of bootstrap, else escalates/self-terminates — mirror into.claude/agents/experimenter.md's launch contract.
Provenance
-
fingerprint: 1e7fcd03e307
-
workflow_fix_target: .claude/rules/pod-side-reporting.md, .claude/agents/experimenter.md
-
origin: /daily 2026-08-04 recovery sweep — miner 4 P1/P2 (probed at discovery).