workflow-fix: scope preflight branch-freshness fetch, raise timeout
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by the workflow-fix-on-bug protocol from a gap the orchestrator hit
directly while running the mandatory pre-launch preflight on pod-2091
(task #2091). Fleet-wide: it fails EVERY feature-branch pod preflight, not just
this one.
Goal
Scope the preflight branch-freshness fetch to the current branch
(git fetch origin <branch>) and raise its timeout, so a repo with many remote
refs cannot make the check unsatisfiable.
Workflow gap
- Bug observed:
src/explore_persona_space/orchestrate/preflight.py:355runs the FULLgit fetch --quiet originwithtimeout=15. The repo now carries 1,633 remote branches (staleissue-*branches accumulate), and a full fetch measured 192 s onpod-2091(rc=0 — it SUCCEEDS, it is only slow; a warm re-run still exceeds 15 s on ref negotiation alone). Because_check_feature_branch_behindtreats a failed fetch as a hard ERROR (while_check_main_branch_behindand_check_detached_head_behindboth deliberately fail-SOFT to a WARNING on the same condition), every preflight run on anissue-<N>branch now reportsPre-flight Check: FAIL / git fetch origin failed (timeout) — cannot verify branch <b> is up to date with origin/<b>even when the branch is provably at0 ahead / 0 behind. - Why it is a workflow gap: the pre-launch preflight is MANDATORY for every
experimenter launch (CLAUDE.md § Pre-launch protocol step 2, "Fix any failure
— don't skip"), so a check that cannot pass on any feature branch either
blocks every pod launch or trains every session to wave the FAIL through —
which then hides the REAL failures the same gate exists to catch (GPU
contention, env drift, EDQUOT headroom). The fetch is also doing far more
work than the check needs: the check only compares HEAD against
origin/<current-branch>, so fetching all 1,633 refs is pure waste. A targetedgit fetch origin <branch>completed in seconds on the same pod in the same session. - Confidence (emitter): high
- verified-at-filing:
grep -n 'fetch", "--quiet", "origin"\], timeout=15' src/explore_persona_space/orchestrate/preflight.py→ 1 hit (line 355) in the single named target file (2026-08-05);git ls-remote --heads origin | wc -l→ 1633 (VM-side, same value read pod-side); timed pod-side full fetchrc=0 elapsed=192sunder a 600 s cap; targetedgit fetch origin issue-2091completed in seconds; the fail-soft-vs-ERROR asymmetry read directly from_check_main_branch_behind/_check_detached_head_behind(WARNING) vs_check_feature_branch_behind(ERROR) in the same file.
Proposed change (candidate diff sketch — refine in planning)
- fetch_rc, _, fetch_err = _run(
- ["git", "-C", str(project_root), "fetch", "--quiet", "origin"], timeout=15
- )
+ # Scope to the branch actually being verified: the freshness check only
+ # compares HEAD against origin/<branch>, so fetching every remote ref is
+ # unnecessary work that scales with branch count (1,633 refs -> ~192 s).
+ _fetch_cmd = ["git", "-C", str(project_root), "fetch", "--quiet", "origin"]
+ if branch and branch != "HEAD":
+ _fetch_cmd.append(branch)
+ fetch_rc, _, fetch_err = _run(_fetch_cmd, timeout=90)
Two dependent details for the planner to resolve, NOT settled here:
branchis currently resolved AFTER this fetch call in the same function — the branch-name resolution must move above the fetch (or the fetch must be deferred) for the targeted form to work. Verify the ordering rather than assuming it.- A targeted fetch updates only
refs/remotes/origin/<branch>, so theorigin/maincomparisons that follow (_behind_count(project_root, "origin/main")in both the main-branch and detached-HEAD paths, plus the feature-branch informational divergence WARNING) would read a possibly-stalerorigin/main. Either fetch both refs (git fetch origin <branch> main) or keep theorigin/maincomparison explicitly fail-soft and say so in the summary line. Do not silently degrade an existing check.
The timeout value (90 s above) is a placeholder — ground it on a measured targeted-fetch time across lanes (RunPod pod, GCE, the shared VM), not on this sketch.
Scope / surfaces
- Primary target:
src/explore_persona_space/orchestrate/preflight.py - Also likely:
tests/test_preflight_disk.pyor a sibling preflight test — add a regression pin that a feature-branch preflight does NOT hard-FAIL when the branch is in sync but a full-remote fetch would be slow (fake the fetch seam). - Grep the workflow surface for other unscoped full-remote fetches under a
short timeout before editing
(
grep -rn 'fetch", "--quiet", "origin"' src/ scripts/), and list every hit in the plan.
Constraints / invariants
- Do NOT weaken the feature-branch behind/diverged ERROR itself — a branch that is genuinely behind its own origin ref must still hard-FAIL. The fix is to make the fetch that FEEDS that check reliable, not to demote the check.
uv run python scripts/workflow_lint.pypasses; ruff on touched files passes.- The cluster branch already SKIPS the fetch round trip
(
report.git_status += " (cluster — skipped fetch)") — keep that path byte-unchanged.
Related concern (separate, do not bundle)
1,633 remote branches is itself worth a look — stale issue-* branches are
never pruned after their Step 10d merge, and every clone/fetch pays for them.
That is a distinct hygiene question with a different target surface (a branch
janitor, likely alongside worktree_audit.py / cron_worktree_audit.sh) and
should be filed on its own rather than folded into this fix.
Provenance
- workflow_fix_target: src/explore_persona_space/orchestrate/preflight.py
- fingerprint: 109b2c35251b
Raised directly by the orchestrator of task #2091 while running the mandatory
pre-launch preflight on pod-2091 (4x H100). The run proceeded on an
independently verified invariant (git rev-list --left-right --count HEAD...origin/issue-2091 → 0 0, plus a full fetch confirmed rc=0), with the
disposition recorded in an epm:progress marker on #2091.