workflow-fix: GCP gate-park teardown leg + terminal-phase reconnect refusal
kind: infra#wf-fix#wf-fix-fp:3cbb26c26d52
track:Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by the workflow-fix-on-bug protocol from the orchestrator's own incident on task #763 (cofit round, 2026-07-03).
Goal
Add a GCP-lane instance-teardown leg to the /issue Step 6d.4 gate handlers (and the gate-handler authoring guidance): on a BLOCKING gate emitted by a GCP-lane workload, the orchestrator deletes the instance after the sentinel is drained and the gate is resolved — and the router's reconnect path must never treat a gate-parked/terminal-phase instance as a live reconnect target.
Workflow gap
- Bug observed (two legs, one incident): (1) #763's
cofit_phaseA_donegate-park kept the GCE instance RUNNING after[phase=done]— correct for sentinel draining, but no handler step tears it down; the orchestrator resolved the gate (launched the off-pod phase) without deleting the instance, and an A100-80 idled ~40 min. (2) The NEXTdispatch_issue.py launchfor the same issue returnedreason: reconnectto that zombie (RUNNING, terminaleps/phase=workload_done) WITHOUT launching the new workload — silently no-op'ing the phase-C dispatch until the orchestrator noticed the poll readingworkload_done. - Why it is a workflow gap: Step 6d.4 documents the RunPod pod-cycling teardown (
pod.py stop) forpv_phase1_donebut has no GCP-lane analogue;gcp.reconnect_or_nonetreats any RUNNINGeps-issue-<N>as reconnectable without checking the workload phase is non-terminal. Both legs will recur for every multi-phase GCP-lane experiment that gates mid-pipeline. - Confidence (emitter): high (reproduced live on #763; markers 03:41-03:52Z).
Proposed change (candidate diff sketch — refine in planning)
.claude/skills/issue/SKILL.md, Step 6d.4:
+ GCP-lane blocking gates: after the poller drains the sentinel and the
+ handler resolves the gate (parks OR auto-resolves), if the handle's
+ backend is gcp and the workload printed [phase=done]/a terminal gate
+ park, DELETE the instance (gcloud compute instances delete, or
+ dispatch_issue.py finalize --skip-confirm-artifacts when mid-pipeline)
+ BEFORE dispatching the off-lane phase — the instance stays up ONLY for
+ sentinel draining, never through an off-pod phase (8-bis).
src/explore_persona_space/backends/gcp.py reconnect_or_none:
+ refuse to reconnect to an instance whose eps/phase guest attribute is
+ terminal (workload_done / a blocking-gate park) — return None so the
+ router creates a fresh instance.
Scope / surfaces
- Primary targets:
.claude/skills/issue/SKILL.md(Step 6d.4),src/explore_persona_space/backends/gcp.py(reconnect predicate — in-scope backends module). - Grep:
grep -rn "reconnect" src/explore_persona_space/backends/gcp.py scripts/backend_poll.py; check the janitor's zombie predicate (gcp_audit.py) for the same RUNNING-but-terminal-phase signature to share it.
Constraints / invariants
- Workflow-surface only; router tests (
tests/test_router*.py,tests/test_gcp_backend.py) stay green; add a pin for the terminal-phase reconnect refusal.
Provenance
- workflow_fix_target: .claude/skills/issue/SKILL.md, src/explore_persona_space/backends/gcp.py
- fingerprint: 3cbb26c26d52
Activity