daily-fix: pod-side liveness probe before LAUNCH FAILED
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Workflow gap
Two same-day #1739 fan-out patterns wrote false negatives into durable/driver records because the reporting layer keyed on the SSH channel instead of pod-side state:
- Launch-wrapper false "LAUNCH FAILED". The 6-pod jobd/r2aug fan-out's launch ssh hung (~35 min for jobd-evil), was killed, returned non-zero — and the wrapper wrote "pod-1739-jobd-evil LAUNCH FAILED" while the detached pod-side run was alive (pid 1885 verified). Recurred for jobd-sycophancy at 07:48Z. The teammate shipped a session-local v2 driver with a 60 s launch-watchdog that probes the pod-side leg before reaping hung sshs — the recipe exists but lives only in that session. A sibling instance the same day: pod-1739-a1apilot "LAUNCH FAILED (ssh_rc=124, no live pod-side proc)" needed a manual re-pin + relaunch (that one was a REAL failure — the probe is what tells them apart).
- Heartbeat false "UNREACHABLE — investigate". The R5 wall-clock heartbeat flagged terminated-after-verified-upload pods as "UNREACHABLE (ssh rc=255) — investigate before assuming healthy" twice (07:45Z, 10:01Z); the 10:02Z heartbeat corrected the copy to "TERMINATED (verified upload) … NOT a fault" by checking the termination sentinel first.
verified-at-filing: all quotes are probed tool_result/driver-log readbacks from session
21e049f7 (rows 4113–4260, 4222/4556/4569) and 2f4940f0 (rows 271–328).
grep -n 'LAUNCH FAILED\|liveness' .claude/rules/pod-side-reporting.md | head run at
compose time — the (re)launch contract covers pid-file/log/sentinel duties but has no
"probe before declaring failure" clause for wrappers/heartbeats.
Proposed change
Add to .claude/rules/pod-side-reporting.md (the (re)launch contract section): a launch
wrapper never writes LAUNCH FAILED without a pod-side pidfile/process probe (the 60 s
launch-watchdog pattern: ssh channel outcome ≠ leg outcome), and a heartbeat classifies
ssh rc=255 against the termination sentinel / pod.py list-ephemeral before writing
UNREACHABLE. Name the v2-driver watchdog as the reference implementation so the next
fan-out inherits it instead of re-deriving it.
Provenance
-
fingerprint: 068a96affa5c
-
workflow_fix_target: .claude/rules/pod-side-reporting.md
-
origin: /daily 2026-08-05 problem sweep — miner 2 P2/P21.