issue: 751 kind: infra architectural: false tdd: no estimated_gpu_hours_total: 0 plan_version: 1 plan_at_sha: 7f0ec39a0ef7ae78ee9d6685b6d0c68a11522d54

Plan — #751 failover relaunch must register the new pod in pods.conf; recovery commands must create-missing

0. Plan Summary

What: Close a three-link recovery failure that bit pod-664 / pod-697 / pod-658. When a GCP→RunPod failover (or a RunPod no-port-wedge re-provision) relaunches a pod, that pod can end up absent from pods.conf (the SSH/MCP config source); the SSH poller then reads status=dead on a transport failure even though the pod is RUNNING per the live API; and the documented manual recovery (pod.py config --update / --refresh-from-api) refuses with "pod not found in pods.conf" because the row never existed.

Three additive fixes (no CLI/marker/enum/contract change):

scripts/backend_poll.py — after a successful RunPod failover relaunch (both _failover_dead_gcp_to_runpod and the no-port-wedge _relaunch_fresh_runpod), make a best-effort, fail-soft call that pulls the new pod's live host/port into pods.conf (via the create-missing --refresh-from-api from fix 2). The failover already shells pod_lifecycle.py provision (which calls _upsert_pods_conf), but that registration is bypassed when provision hits the idempotency refusal or the SSH-wait window expires — so this is the belt-and-suspenders re-registration.
scripts/pod_config.py — cmd_update (line 707) and cmd_refresh_from_api (line 911) CREATE a missing pods.conf entry from the live RunPod API (runpod_api.list_team_pods()) instead of sys.exit(1) "pod not found in pods.conf" (lines 726/963). Respect manual_override: a --update-created row is user-pinned (override=True, matching the update-existing path); a --refresh-from-api-created row is API-sourced (override left False).
scripts/poll_pipeline.py — on a name-resolution SSH failure SPECIFICALLY (ssh: Could not resolve hostname, classified distinctly from connection-refused / timed-out), poll_once cross-checks the live RunPod API before reporting status=dead. If the live API says the pod is RUNNING, it rehydrates pods.conf via the create-missing --refresh-from-api (fix 2) and re-probes ONCE; a successful re-probe avoids the false dead verdict. Real connection-refused / timeout failures (the ssh_error_class == "other" case) are UNTOUCHED — they still route to dead via the existing path. This is the EARLIER, name-resolution-specific complement to the existing #488 10-consecutive-failure auto-heal.

Baselines / controls: N/A — infra fix. Hyperparameters / Loss / Eval: N/A — no model. Compute: 0 GPU-hours; CPU unit tests only.

Risks (top 2): (a) mis-factoring the create-missing path could weaken the existing manual_override protection (mitigated: create-path is a separate branch from the update-existing path; existing override tests must stay green); (b) fix 3's live-API cross-check must stay strictly scoped to the ssh_error_class == "name_resolution" case and require a SUCCESSFUL re-probe (a real dead PID with healthy SSH, and a connection-refused/timeout pod, must still report dead) and must not add an unbounded per-tick API call (mitigated: gate on the name-resolution class only, re-probe once, fail-soft to the unchanged dead path on any API error / non-RUNNING live pod / still-failing re-probe — never on the healthy-SSH PID-probe-dead path or the "other" SSH-failure class).

Estimated GPU-hours (total): 0

1. Goal

Verbatim from body: "A failover/relaunched pod is always registered in pods.conf, and the documented recovery commands work when the pod is absent."

Acceptance criteria (binary):

A1 — After a successful RunPod failover relaunch, the new pod's row is present in pods.conf (or a fail-soft re-registration was attempted and logged). Verified by a unit test asserting the re-registration call fires on the success path of _failover_dead_gcp_to_runpod.
A2 — pod.py config --update <pod> on a pod ABSENT from pods.conf CREATES the row from the live API + sets manual_override=True, instead of exiting 1. Verified by a unit test.
A3 — pod.py config --refresh-from-api <pod> on a pod ABSENT from pods.conf CREATES the row from the live API (leaving manual_override False), instead of exiting 1. Verified by a unit test.
A4 — poll_once does NOT report status=dead when SSH fails (ssh_failed=1, pid_alive=0) but the live RunPod API reports the pod RUNNING; it reports running (degraded-observability, the existing ssh_failed self-heal regime). Verified by a unit test. A genuinely dead pod (PID probed dead with healthy SSH, OR API confirms not-RUNNING) still reports dead.
A5 — uv run ruff check + uv run ruff format clean on touched files; existing tests/test_pod_config*.py, tests/test_backend_poll*.py, tests/test_poll_pipeline_sentinels.py, tests/test_router*.py, tests/test_no_auto_runpod_path_under_any_failure.py all stay green.

2. Prior Work

The recovery surface this touches was built incrementally by the #488 stale-port line:

59118210ac / 3fa509e59e — introduced pod.py config --refresh-from-api (#488 stale-port recovery: pull live host/port from the RunPod API into pods.conf). The single-pod mode of this command is exactly where fix 2 adds create-missing.
2a554be9c0 / 2352dc8dc4 — auto-fire pod.py config --refresh-from-api from the SSH-poll staleness counter (the _try_refresh_pods_conf_from_api helper in poll_pipeline.py, threshold SSH_FAIL_REFRESH_THRESHOLD=10) and the session-watch stalled-RESPAWN branch (a82856a24c).
05545494c9 — 1h SSH-wait alarm on a billing pod (pod_lifecycle.py + poll_pipeline.py, refs #572).
29658492aa — authority-split mechanics → .claude/rules/pod-config.md (live API authoritative for state; pods.conf the SSH/MCP source; the three sync directions).

The failover paths themselves were built by:

#659 (_failover_dead_gcp_to_runpod, async GCP→RunPod failover) and #689/#664 (_relaunch_fresh_runpod, RunPod no-port-wedge re-provision). Both reconstruct a RunSpec and call failover_to_runpod_after_async_workload_crash → RunPodBackend().launch(spec).

The bug is the seam between these two lines. The #488 self-heal already calls --refresh-from-api from the poller — but it errors out when the pod is absent (the state a failover leaves), so the auto-heal can never recover a failover-relaunched pod. This is captured live in the recovery trail: git log shows e36d598d43 (#664) — "pod-664 missing from pods.conf — recovered" — the exact incident this task closes.

The authority split (.claude/rules/pod-config.md) is the governing invariant: the live RunPod API is authoritative for pod state; pods.conf is the derived SSH/MCP config. Create-missing from the live API is the natural extension of "Live API → pods.conf", already implemented by --refresh-from-api for the update-existing case.

3. Method Delta (vs. the existing provision/resume registration path)

pod.py provision / resume register a pod through pod_lifecycle.py::_upsert_pods_conf (line 448), which under locked_pods_conf(): parses pods.conf, finds-or-appends the row, writes, and calls cmd_sync. It is fed an EphemeralPod carrying host/port from the live wait_for_ssh result.

What differs at failover: RunPodBackend.launch (in src/explore_persona_space/backends/runpod.py) shells out to pod_lifecycle.py provision, whose tail _provision_wait_register_bootstrap DOES call _upsert_pods_conf (line 1658) — BUT that registration is skipped in two real failover situations:

Idempotency refusal (cmd_provision, line 1702): if a non-EXITED pod for the issue already exists (the wedge case where the old pod is still RUNNING-but-no-port, or a race), provision exits 1 before reaching _upsert_pods_conf.
SSH-wait expiry (_provision_wait_register_bootstrap, line 1641): if the pod is created+billing but exposes no public 22/tcp within 600s, the function raises and explicitly prints "pods.conf was NOT updated ... run pod.py config --refresh-from-api" — but for a NEW pod that row does not exist, so today that advice fails.

The minimal, non-duplicating delta is therefore NOT to re-implement _upsert_pods_conf inside the failover path (the launch subprocess already attempts it on the happy path) but to:

add a fail-soft post-launch re-registration in the two backend_poll.py failover success paths that calls the SAME create-missing --refresh-from-api recovery (fix 2), AND
make --refresh-from-api / --update create-missing (fix 2) so that re-registration — and the existing #488 poller self-heal — actually works for an absent pod.

This keeps ONE registration story (_upsert_pods_conf for the create-time-known-host case; the live-API create-missing branch for the recovery case) and reuses the already-built --refresh-from-api plumbing rather than inventing a parallel path. The new shared helper in fix 2 is a small _pod_from_live(name, live) that builds a Pod from a PodInfo, called by both create branches.

4. Design (per fix)

Fix 2 (do this FIRST — fixes 1 and 3's self-heal both depend on it)

File: scripts/pod_config.py.

New shared helper (place near _refresh_one_pod, ~line 818):

def _pod_from_live(name: str, live: PodInfo) -> Pod:
    """Build a pods.conf Pod row from a live RunPod PodInfo.

    Used by the create-missing branches of cmd_update / cmd_refresh_from_api.
    Caller MUST have verified live.ssh_host/ssh_port are populated (a
    non-RUNNING / no-port pod has no SSH endpoint to record).
    """
    assert live.ssh_host is not None and live.ssh_port is not None, (name, live)
    return Pod(
        name=name,
        host=live.ssh_host,
        port=live.ssh_port,
        gpus=live.gpu_count or 1,
        gpu_type=_short_gpu_label_from_live(live),  # reuse existing label derivation if present
        label=name,                                 # human label; refined on next provision
    )

gpu_type is cosmetic for SSH/MCP generation (_ssh_entry/_generate_mcp_env use only name/host/port); derive a short label from live.gpu_type_id if a reusable mapping is reachable in pod_config, else store the raw gpu_type_id or "unknown". Implementer's call (§9).

cmd_update create-missing (replace the sys.exit(1) at line 726): when pod_name is absent from the parsed pods, do NOT exit. Inside locked_pods_conf() re-parse fresh; if pod_name is STILL absent, query the live API (from runpod_api import list_team_pods):

The --update contract REQUIRES at least one of --host/--port (line 721) — preserve that precondition. Build the row from the user-supplied --host/--port, filling gpus/gpu_type from the live API entry when present (_pod_from_live then overlay the user values), else placeholders (gpus=1, gpu_type="unknown"). Append to fresh, set manual_override=True (a --update-created row is user-pinned, consistent with the existing _set_manual_override(pod_name, value=True) at line 764), write + cmd_sync, print "Created pod 'X' in pods.conf from user-supplied values (+ live API fill-ins)".
This always succeeds (the user supplied the endpoint), so there is no remaining hard-error in cmd_update's create path.

cmd_refresh_from_api create-missing (replace the sys.exit(1) at lines 962-965, single-pod mode only): when pod_name is absent from pods, do NOT exit. Use the already-fetched live_by_name (line 954):

pod present in live API AND RUNNING AND has ssh_host/ssh_port: build the row via _pod_from_live, append it to the targets inside the lock (leave manual_override False — API-sourced, not user-pinned), let the existing per-pod write + cmd_sync run, print "Created pod 'X' in pods.conf from live API".
pod absent from live API, OR not RUNNING, OR no SSH endpoint: keep the existing hard-error (reworded to name the live-API miss) — --refresh-from-api cannot fabricate an SSH endpoint that the platform has not assigned (fail-fast, per CLAUDE.md). Bulk mode (pod_name=None) is unchanged — it only iterates pods already in pods.conf.

manual_override interaction (explicit): create-missing produces a NEW row, so there is no prior on-disk value to protect. --update-create sets manual_override=True; --refresh-from-api-create leaves it at default (absent → False). _set_manual_override returns None and does NOT write when pods_ephemeral.json lacks the pod (line 696-697 / 683-684) — so a --update-created row whose pod has no sidecar entry yet cannot have its override flag persisted. Known limitation (acceptable): the row IS created (recovery + SSH work — the PRIMARY goal), and override-protection engages once a later provision/resume registers the pod in pods_ephemeral.json (provision creates that entry anyway). Do NOT auto-create a sidecar entry from --update — that would expand pods_ephemeral.json's ownership semantics (provision/resume own it). Document this in the code comment.

Fix 1 (depends on fix 2)

File: scripts/backend_poll.py.

In BOTH RunPod-failover success paths — _failover_dead_gcp_to_runpod (just before the final return {"status": "running", ...} at ~line 1695) and _relaunch_fresh_runpod (just before its success return) — add a fail-soft re-registration of the new RunPod pod into pods.conf. The new pod's host/port are NOT on the RunHandle (it carries pod_name; host/port are populated inside pod_lifecycle.py and read from the live API), so re-registration MUST go through the live API. Reuse the create-missing --refresh-from-api (fix 2):

def _register_failover_pod_in_pods_conf(pod_name: str) -> None:
    """Fail-soft: pull the just-launched failover pod's live host/port into
    pods.conf (+ SSH/MCP) so the documented recovery commands and the SSH
    poller can find it. The launch subprocess (pod_lifecycle provision)
    attempts _upsert_pods_conf on its happy path; this is the belt-and-
    suspenders re-registration for the idempotency-refusal / SSH-wait-expiry
    cases where that upsert was skipped. NEVER raises (a failover that
    succeeded must not be reported failed because a cosmetic config sync
    hiccuped) — logs a warning and returns.
    """
    cmd = [sys.executable, str(_scripts_dir / "pod.py"),
           "config", "--refresh-from-api", pod_name]
    try:
        subprocess.run(cmd, capture_output=True, text=True, timeout=60, check=False)
    except (OSError, subprocess.TimeoutExpired) as exc:
        log.warning("failover pods.conf re-register for %s raised %s; "
                    "recovery via `pod.py config --refresh-from-api %s` may be needed",
                    pod_name, type(exc).__name__, pod_name)

Call it with route_result.handle.pod_name right after the authoritative sidecar readback confirms recovered.backend == "runpod", before the success return. Invoke ONLY on the genuine-success branch (NOT on the idempotency short-circuit, NOT on any terminal-infra-JSON return, NOT on the M3b concurrent-triggerer short-circuit — in that branch the OTHER triggerer's launch already re-registers). Mirror the existing _try_refresh_pods_conf_from_api fail-soft shape in poll_pipeline.py. Resolve _scripts_dir via whatever path helper backend_poll.py already uses to locate scripts/ (it lives IN scripts/, so Path(__file__).resolve().parent is the fallback — grep the file for an existing pod.py/task.py subprocess call site first).

Why a subprocess, not an in-process call: pod_config.cmd_refresh_from_api is a CLI command taking a list[Pod] that main parses; the subprocess keeps lock-discipline + arg-parsing in one place and matches the already-shipped _try_refresh_pods_conf_from_api self-heal pattern (CLAUDE.md "Reuse existing in-repo tools"). An in-process import would re-derive the parsed-pods arg and duplicate main's plumbing.

Fix 3 (depends on fix 2 for the rehydrate step; together they form the recovery chain)

Files: scripts/poll_pipeline.py — _ssh_probe (~line 1104) + poll_once (~line 2319 probe call, verdict at line 2497).

The brief is explicit that fix 3 must (a) act on a name-resolution SSH failure SPECIFICALLY (distinct from connection-refused / timed-out), and (b) re-probe ONCE after rehydrating pods.conf. A pod whose host pods.conf does not yet know surfaces as ssh: Could not resolve hostname pod-<N> — that is the exact signature the rehydrate-and-retry fixes; a real Connection refused / Connection timed out means the address IS known and the pod is genuinely unreachable → still dead.

Step 1 — classify the SSH failure in _ssh_probe. Today _ssh_probe logs but DISCARDS result.stderr (line 1105) and returns only ssh_failed. Surface the failure class so poll_once can branch without re-running ssh:

if result.returncode != 0:
    stderr = result.stderr or ""
    log.error("ssh failed (rc=%d): %s", result.returncode, stderr.strip())
    name_unresolved = "could not resolve hostname" in stderr.lower()
    return {
        ...,                                  # existing zeroed fields unchanged
        "ssh_failed": "1",
        "ssh_error_class": "name_resolution" if name_unresolved else "other",
    }
parsed = _parse_probe_stdout(result.stdout)
parsed["ssh_failed"] = "0"
parsed["ssh_error_class"] = "ok"
return parsed

(Additive key; existing probe.get(...)/probe[...] reads in poll_once index by their own keys and are untouched — Assumption 10 / §14.)

Step 2 — name-resolution heal in poll_once, immediately after the probe = _ssh_probe(...) call (~line 2319) and BEFORE _update_ssh_fail_tracking:

probe = _ssh_probe(pod, log_path, pid_file, issue, marker_pid)
if probe.get("ssh_failed") == "1" and probe.get("ssh_error_class") == "name_resolution":
    healed = _try_heal_name_resolution(pod, issue, log_path, pid_file, marker_pid)
    if healed is not None:
        probe = healed     # re-probe after rehydrate succeeded -> not dead this tick
    # else: fall through; the existing 10-fail auto-heal + the line-2497 dead path apply

The elif not pid_alive: status = "dead" verdict at line 2497-2498 is then UNCHANGED — a successful heal swaps in the healthy re-probe (so pid_alive is True and the dead branch is not taken); an unsuccessful heal (or the "other" class) leaves the failed probe in place and the existing path runs verbatim.

New fail-soft helper near _try_refresh_pods_conf_from_api (~line 516):

def _try_heal_name_resolution(pod, issue, log_path, pid_file, marker_pid) -> dict | None:
    """On an ssh name-resolution failure, cross-check the live RunPod API; if the
    pod is RUNNING, rehydrate pods.conf via --refresh-from-api (create-missing,
    fix 2) and re-probe ONCE. Returns the re-probe dict on success, or None to
    fall through to the unchanged dead path. Fail-soft: any API/subprocess error,
    a non-RUNNING / absent live pod, or a still-failing re-probe -> None."""
    try:
        from runpod_api import get_pod_by_name
        info = get_pod_by_name(pod)
    except Exception as exc:   # noqa: BLE001 — fail-soft; must not crash the poll loop
        log.warning("name-resolution heal: live-API check for %s raised %s; "
                    "falling through to dead verdict", pod, type(exc).__name__)
        return None
    if info is None or (info.desired_status or "").upper() != "RUNNING" \
       or not info.ssh_host or not info.ssh_port:
        return None
    # Live API says RUNNING with an SSH endpoint -> pods.conf is just missing/stale.
    # Reuse the EXISTING fail-soft --refresh-from-api wrapper (now create-missing).
    if not _try_refresh_pods_conf_from_api(pod):
        return None
    reprobe = _ssh_probe(pod, log_path, pid_file, issue, marker_pid)
    return reprobe if reprobe.get("ssh_failed") == "0" else None

It reuses the EXISTING _try_refresh_pods_conf_from_api(pod) (line 516 — the same fail-soft, 60s-timeout pod.py config --refresh-from-api <pod> wrapper the #488 auto-heal already uses), which after fix 2 will CREATE the missing row. No new subprocess wrapper.

Cost / call-rate: the get_pod_by_name live-API call fires ONLY on the ssh_error_class == "name_resolution" branch (exceptional — a healthy run never hits it, and a connection-refused/timeout run takes the "other" path with no API call), so it adds at most one live-API call + one re-probe per name-resolution-failing tick. Same list_team_pods-class call the #488 self-heal already makes.

except Exception justification (vs the fail-fast rule): this is a fail-SOFT cross-check whose explicit contract is "on any uncertainty, fall back to the pre-fix dead verdict" — it never swallows a real death (a genuine API outage returns False → dead, the conservative direction) and is logged loud (not silent), matching the established _try_refresh_pods_conf_from_api fail-soft pattern in the same file. This is the documented exception class (fail-soft recovery helper), not the banned silent-swallow. A narrower (OSError, RunPodError) tuple is acceptable if the implementer prefers (the RunPodError import is cheap) — implementer's call (§9).

5. Conditions and Controls

Plain-English name	What it tests	What it controls for	Config slug
Failover re-registers pod	A successful GCP→RunPod failover calls the pods.conf re-register	the idempotency-refusal / SSH-wait-expiry registration gap	`_failover_dead_gcp_to_runpod` success path
Wedge re-launch re-registers	A no-port-wedge re-provision calls the same re-register	the wedge sibling of the same gap	`_relaunch_fresh_runpod` success path
Update creates missing	`--update <absent>` creates the row from user values + live fill-ins, sets override	the "pod not found" hard-exit at line 726	`cmd_update`
Refresh creates missing	`--refresh-from-api <absent>` creates the row from the live API	the "pod not found" hard-exit at line 963	`cmd_refresh_from_api`
Name-resolution heal	name-resolution SSH-fail + live-RUNNING → rehydrate + re-probe once, not `dead`	the false-`dead` on a stale-port / unknown-host pod	`poll_once`
Other SSH-fail still routes normally (control)	connection-refused / timeout (`ssh_error_class="other"`) is NOT cross-checked, routes to `dead` unchanged	over-broadening fix 3 beyond name-resolution	`poll_once`
Real-dead still dead (control)	PID probed dead with healthy SSH (or API not-RUNNING / API error) still `dead`	over-narrowing fix 3 into masking real deaths	`poll_once`

6. Evaluation

N/A — no behavioral construct measured. Success is the binary acceptance criteria A1–A5 (§1), each a unit test. No DV, no judge, no statistics.

6.5 Primary deliverable

primary_deliverable: []
# N/A — kind: infra workflow-fix; no on-pod primary artifact. Success is the
# passing unit-test suite (§10) + clean ruff, verified at the Step 9c
# test-verdict gate, not an eval_results/ artifact.

7. Decision Gates

No gates — 0 GPU-h, single-round infra fix; no run to stop early.

8. Risks and Failure Modes

Risk	Likelihood	Mitigation
Create-missing weakens existing `manual_override` protection	Low	Create is a SEPARATE branch from update-existing; the override-preservation tests (`test_pod_config_sync_preserves_manual.py`, `test_refresh_respects_manual_override`) must stay green. `--update`-create sets override=True; `--refresh`-create leaves it False — matching each command's existing convention.
Lock-discipline regression	Low	All create-missing writes happen INSIDE the existing `locked_pods_conf()` block, re-parsing `fresh` under the lock (as `cmd_update`/`cmd_refresh_from_api` already do at lines 730/974). No new lock; no write outside it. `test_pod_config_locking.py` must stay green.
Fix 3 masks a real dead workload	Low-Med	Gated strictly on `ssh_error_class == "name_resolution"` AND a SUCCESSFUL re-probe; fail-soft returns None (→ unchanged `dead` path) on ANY API error, a non-RUNNING/absent live pod, or a still-failing re-probe; a healthy-SSH dead PID never reaches the heal (its `ssh_error_class == "ok"`); a connection-refused/timeout pod takes the `"other"` path untouched. Control tests (other-still-routes-normally, real-dead-still-dead) pin this.
Fix 3 adds per-tick API pressure	Low	The `get_pod_by_name` call fires only on the rare `name_resolution` branch, not every tick (and not on the `"other"`/healthy paths); same `list_team_pods`-class call the #488 self-heal already makes.
Name-resolution string match too broad/narrow across ssh versions	Low	Match the stable OpenSSH phrase `could not resolve hostname` case-insensitively; the `"other"` class is the safe default (routes to `dead`, unchanged) so a missed match degrades to today's behavior, never worse.
Fix 1 reports a succeeded failover as failed	Low	`_register_failover_pod_in_pods_conf` NEVER raises (fail-soft); called after the authoritative sidecar readback, only on the success branch.
Breaking the no-auto-RunPod invariant	Low	No routing change; `tests/test_no_auto_runpod_path_under_any_failure.py` + `test_router*.py` must stay green. Fixes are pure registration/observability, downstream of routing.
`--update`-created row's override flag not persisted (no sidecar entry)	Low (known limitation)	Documented in §4: the row IS created (PRIMARY goal met); override engages once provision/resume registers the pod in `pods_ephemeral.json`. Do NOT expand sidecar ownership from `--update`.

9. Resources & Parallelism

0 GPU-hours. CPU-only unit tests on the dev VM. No pod, no API spend (tests stub list_team_pods). No parallelism axis applies — three small edits + tests, landed in one implementer round.

Plan-deviation latitude (implementer's call): the helper names (_pod_from_live, _register_failover_pod_in_pods_conf, _try_heal_name_resolution), the gpu_type short-label derivation, the _scripts_dir path-helper reuse, the exact fail-soft exception tuple in fix 3 (except Exception fail-soft vs a narrower (OSError, RunPodError)), whether the fix-3 tests extend tests/test_poll_pipeline_sentinels.py or land in a new tests/test_poll_pipeline_name_resolution_heal.py, and whether fix-2 create-missing is done via a create_missing param on _refresh_one_pod vs pre-seeding fresh_by_name. MUST come back to the user (architectural / contract — re-plan): switching fix 1 to the more-invasive "upsert before SSH-wait" restructuring of _provision_wait_register_bootstrap; any change to manual_override SEMANTICS beyond "create sets it per the command's existing convention"; any new CLI flag/subcommand; any change to the pods.conf 6-field format; any change to the poller's dead/running/stalled status enum or any marker schema; touching any out-of-scope surface (src/explore_persona_space/{train,eval,axis,analysis}, configs/, tasks/).

10. Tests

All CPU, all extend existing files. Exact invocations:

Fix 2 — tests/test_pod_config_refresh_from_api.py (reuse the existing stub_list_team_pods / stubbed_pods_conf / isolated_sidecar fixtures, lines 111-186):

test_refresh_creates_missing_pod_from_live_api — pod absent from pods.conf, present + RUNNING + has ssh_host/port in the stub → row CREATED, cmd_sync called, manual_override left False, exit NOT 1.
test_refresh_missing_pod_not_in_live_api_still_errors — absent from BOTH pods.conf and live API → hard-error preserved (cannot fabricate an endpoint).
test_refresh_missing_pod_not_running_still_errors — absent from pods.conf, present but desired_status=EXITED → hard-error preserved.
Run: uv run pytest tests/test_pod_config_refresh_from_api.py -x

Fix 2 — tests/test_pod_config.py (add cmd_update create-missing coverage; lift stub_list_team_pods to a shared fixture or duplicate locally):

test_update_creates_missing_pod_from_user_values_sets_override — cmd_update("pod-697", host=..., port=...) with pod absent from pods.conf → row CREATED with the user's host/port, cmd_sync called, exit NOT 1.
test_update_creates_missing_fills_gpu_from_live_api — same, pod RUNNING in live API → gpus/gpu_type filled from the live entry (cosmetic check).
Run: uv run pytest tests/test_pod_config.py -x

Fix 1 — tests/test_backend_poll.py (extend the existing failover tests; the FakeRunPodBackend + write_handle_sidecar + monkeypatch scaffold at lines 97-172 is present):

test_failover_success_reregisters_pod_in_pods_conf — monkeypatch backend_poll._register_failover_pod_in_pods_conf with a spy; assert it is called exactly once with the new pod_name on the success path, and NOT called on the no_compute_available / sidecar_persistence_failed terminal paths (extend test_failover_runpod_unavailable_* / test_failover_sidecar_persistence_failure_*).
test_failover_reregister_is_fail_soft — make the spy raise; assert the failover STILL returns the status=running success JSON (no exception escapes).
Run: uv run pytest tests/test_backend_poll.py -x

Fix 3 — tests/test_poll_pipeline_name_resolution_heal.py (new file mirroring the _try_refresh_pods_conf_from_api + ssh_failed + status==dead patterns in tests/test_poll_pipeline_sentinels.py; or extend that file — implementer's call, §9):

test_name_resolution_fail_with_live_running_reprobes_not_dead — stub _ssh_probe to return ssh_failed=1, ssh_error_class="name_resolution", pid_alive=0 on the FIRST call and a healthy probe (ssh_failed=0, pid_alive=1) on the SECOND; stub get_pod_by_name → RUNNING + ssh-up and _try_refresh_pods_conf_from_api → True; assert poll_once(...).status is NOT "dead" (the re-probe was used).
test_other_ssh_fail_not_cross_checked_routes_dead (control) — ssh_failed=1, ssh_error_class="other" (connection refused); assert get_pod_by_name is NOT consulted and status == "dead" (the "other" path is untouched).
test_name_resolution_fail_live_absent_routes_dead — ssh_error_class="name_resolution" but get_pod_by_name → None (or EXITED) → _try_heal_name_resolution returns None → status == "dead" (conservative fallback).
test_name_resolution_heal_fail_soft_on_api_error — get_pod_by_name raises → helper returns None → status == "dead" preserved (no crash).
test_name_resolution_reprobe_still_failing_routes_dead — heal fires, refresh succeeds, but the re-probe STILL returns ssh_failed=1 → helper returns None → status == "dead".
test_real_dead_pid_with_healthy_ssh_still_dead (control) — ssh_failed=0, ssh_error_class="ok", pid_alive=0 → heal NOT consulted, status == "dead" (fix 3 leaves the healthy-SSH dead path untouched).
Run: uv run pytest tests/test_poll_pipeline_name_resolution_heal.py -x (and tests/test_poll_pipeline_sentinels.py to confirm the additive ssh_error_class field broke nothing).

Regression (must stay green): uv run pytest tests/test_pod_config.py tests/test_pod_config_refresh_from_api.py tests/test_pod_config_sync_preserves_manual.py tests/test_pod_config_locking.py tests/test_backend_poll.py tests/test_poll_pipeline_sentinels.py tests/test_router.py tests/test_no_auto_runpod_path_under_any_failure.py tests/test_runpod_wedge_detection.py

Plus the no-flags workflow lint: uv run python scripts/workflow_lint.py.

11. TDD

### TDD: no

Tests are written alongside the implementation in the same round (the implementer adds the unit tests with each fix), not approval-gated. This is a small, well-scoped infra fix with clear correct behavior; the Step 9c test-verdict gate + the Claude+Codex code-reviewer ensemble are the checks. No epm:proposed-tests/epm:approve-tests cycle.

12. Architectural flag

architectural: false. The three fixes are ADDITIVE behavior — two existing error paths (--update/--refresh-from-api "pod not found" exits) become create-missing, one failover success path gains a fail-soft re-register, one poller verdict gains a live-API guard. NONE change: a task.py/pod.py subcommand or flag, a marker schema, the status enum, the pods.conf 6-field format, or an agent/skill file location. No public contract changes → auto-approve at the 0-GPU-h plan gate, self-merge at Step 10d.

13. Reproducibility Card

Code SHA at plan time: 7f0ec39a0ef7ae78ee9d6685b6d0c68a11522d54 (branch issue-759; this worktree).
Files touched: scripts/pod_config.py, scripts/backend_poll.py, scripts/poll_pipeline.py, plus tests in tests/test_pod_config.py, tests/test_pod_config_refresh_from_api.py, tests/test_backend_poll.py, tests/test_poll_pipeline_sentinels.py.
Verify command: the §10 regression block + uv run ruff check scripts/pod_config.py scripts/backend_poll.py scripts/poll_pipeline.py && uv run ruff format --check scripts/pod_config.py scripts/backend_poll.py scripts/poll_pipeline.py.
GPU-hours: 0. API spend: 0 (tests stub list_team_pods).
TBD (implementer fills): exact merge SHA, final helper names, the gpu_type short-label reuse decision.

14. Assumptions

runpod_api.list_team_pods() returns PodInfo with name, desired_status, ssh_host, ssh_port, gpu_count, gpu_type_id. Confidence: High. Source: read scripts/runpod_api.py:410-449 (PodInfo dataclass + _parse_pod) this session.
cmd_refresh_from_api already lazy-imports list_team_pods and holds locked_pods_conf() for the read-modify-write-sync. Confidence: High. Source: read scripts/pod_config.py:948-999. The create-missing branch slots into the existing lock + live-API fetch (lines 953-954).
RunPodBackend.launch → pod_lifecycle.py provision already calls _upsert_pods_conf on its happy path; the gap is the idempotency-refusal (line 1702) and SSH-wait-expiry (line 1641) skip-paths. Confidence: High. Source: read pod_lifecycle.py:1611-1710 + runpod.py:203-238. This is WHY fix 1 is belt-and-suspenders re-registration, not a missing primary call. (If a reviewer believes the launch subprocess ALWAYS reaches _upsert_pods_conf, fix 1 is harmless redundancy; the incidents prove it does not.)
backend_poll.py lives in scripts/ (NOT src/.../backends/). Confidence: High. Source: clarifier epm:clarify v1-v4 + grep this session. The body's workflow_fix_target path src/explore_persona_space/backends/backend_poll.py is WRONG; the file is scripts/backend_poll.py. (src/.../backends/ has router/runpod/gcp/slurm/issue_dispatch, not the poller.)
The poller's ssh_failed path zeroes pid_alive, so the dead verdict at line 2497-2498 fires on a pure SSH transport failure. Confidence: High. Source: read poll_pipeline.py:1104-1129 (_ssh_probe returns pid_alive="0", ssh_failed="1" on rc!=0) + lines 2497-2498 (elif not pid_alive: status = "dead"). Fix 3 inserts the name-resolution heal BEFORE this verdict (at the probe call ~line 2319) and swaps in a healthy re-probe on success, so the verdict itself is unchanged.
_ssh_probe currently discards result.stderr (logs only) — the ssh: Could not resolve hostname signature is recoverable only by surfacing it. Confidence: High. Source: read poll_pipeline.py:1104-1129 (line 1105 logs result.stderr then returns a dict WITHOUT it). Fix 3 adds an ssh_error_class key to the returned dict ("name_resolution" / "other" / "ok"); this is ADDITIVE — every existing poll_once read indexes the probe dict by its own key (probe["pid_alive"], probe.get("gpu_util"), ...), none iterates keys exhaustively. Verify at implementation: grep -n "probe\[\|probe.get(" scripts/poll_pipeline.py + run tests/test_poll_pipeline_sentinels.py.
OpenSSH emits the stable phrase ssh: Could not resolve hostname <host> for the name-resolution failure, distinct from Connection refused / Connection timed out. Confidence: High (standard OpenSSH client error string). A case-insensitive "could not resolve hostname" in stderr match is the classifier; the "other" default is safe (routes to dead, unchanged) so a missed match never regresses below today's behavior.
runpod_api.get_pod_by_name(name) -> PodInfo | None exists and is team-scoped. Confidence: High. Source: read scripts/runpod_api.py:813-827. Fix 3 uses it (not a full list_team_pods scan) for the single-pod cross-check; PodInfo.desired_status / ssh_host / ssh_port gate the heal.
The #488 self-heal (_try_refresh_pods_conf_from_api, threshold 10) calls --refresh-from-api, which today ERRORS on an absent pod — so the auto-heal cannot recover a failover-relaunched pod until fix 2 lands. Confidence: High. Source: read poll_pipeline.py:516-567 + pod_config.py:962-965. This chain ties fix 2 to fixes 1 and 3.
_failover_dead_gcp_to_runpod and _relaunch_fresh_runpod are the only two RunPod-failover launch success paths in backend_poll.py. Confidence: High. Source: read backend_poll.py (both call failover_to_runpod_after_async_workload_crash). Verify at implementation: grep -n "failover_to_runpod_after_async_workload_crash\|RunPodBackend().launch" scripts/backend_poll.py.
manual_override lives in pods_ephemeral.json, set via _set_manual_override which returns None / does NOT write when the pod is absent from the sidecar (lines 683-697). Confidence: High. Source: read pod_config.py:672-704. Consequence (known limitation, §4/§8): a --update-created row whose pod has no sidecar entry cannot persist its override flag; the row is still created (recovery works), and override engages once provision/resume registers the pod. Verify at implementation: re-read _set_manual_override lines 683-697.
No test currently asserts a failover re-registers pods.conf, nor that --update/--refresh create-missing. Confidence: High. Source: grepped tests/test_pod_config*.py + tests/test_backend_poll.py this session — the create-missing cases hard-exit and have no test. The §10 tests are genuinely new coverage.