workflow-fix: pod-side results-git push must fetch+rebase (mid-run branch-push race)
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by the workflow-fix-on-bug protocol from an orchestrator-observed incident on task #1739 (hallu lane, 2026-07-30 12:19Z).
Goal
Require pod/instance-side results-git pushes to fetch + rebase-onto-origin before pushing (bounded retry), and warn orchestrators that pushing commits to the issue branch while a lane is mid-run races the lane's terminal results-git push into a non-fast-forward workload failure.
Workflow gap
- Bug observed: The hallu lane finished all science (270 cells rc=0, Hub sidecars verified) then exited 1 at the final results-git push: the orchestrator's r10 fix commits advanced origin/issue-1739 past the instance's clone, the driver's push retry detected behind=1 but never fetched/rebased, and the healthy run was classified as a workload crash with poweroff.
- Why it is a workflow gap:
.claude/rules/pod-side-reporting.mdalready owns "pushing result commits" (LESSONS.md row) but is silent on BOTH halves of this race: (a) the pod-side push recipe has no fetch+rebase-before-push requirement, so any orchestrator branch commit mid-run (a crash-fix on a SIBLING lane — the normal multi-lane pattern) deterministically fails the healthy lane's terminal push; (b) nothing warns the orchestrator that a mid-run branch push races live lanes' result pushes. The cost is a full false workload-crash cycle: exit 1 → crash-persist → poweroff → orchestrator forensics + manual result recovery (~30 min today; the science was 31h). - Confidence (emitter): high
- verified-at-filing: incident record on task #1739 epm:progress v89 (2026-07-30T12:50:44Z) + the crash-persist workload.log at HF issue1739_partial/att-20260729-042950-hallu/ lines "[upload] results-git: push attempt 1 rc=1 behind='1' ... ! [rejected] HEAD -> issue-1739 (fetch first)" / "push attempt 2 rc=1 ... (non-fast-forward)" / "push verification FAILED after retry" → "[startup-script] FAILED rc=1".
grep -in "rebase\|fetch" .claude/rules/pod-side-reporting.md→ no fetch/rebase-before-push requirement in the result-push section (2026-07-30). The racing commits: aff188af67df (07:15Z r10 fix push) + 795f474712 (hot-fix) vs the instance clone at 2026-07-29T04:31Z. - unverified hypothesis — verify at plan time: whether the driver's push helper is shared across issue drivers (a shared helper under src/ or scripts/) or per-issue (scripts/issue1739_*.py) — the RULE addition binds either way; a shared-helper code fix would be a separate follow-up in the owning module.
Proposed change (candidate diff sketch — refine in planning)
.claude/rules/pod-side-reporting.md (result-push section)
+ - **Fetch + rebase before every pod-side results-git push.** A lane's
+ terminal push races ANY orchestrator branch commit made mid-run (a
+ sibling lane's crash-fix is the normal case): a bare push retry that
+ detects behind>0 but never fetches loses deterministically —
+ non-fast-forward → workload exit 1 → a HEALTHY run crash-persists and
+ powers off (#1739 hallu: 31h of complete science, recovered manually).
+ Push recipe: git fetch origin <branch> && git rebase origin/<branch>
+ (result commits are additive per-lane files; conflicts are near-
+ impossible) && push, bounded 2 attempts; on rebase conflict fall back
+ to crash-persist WITHOUT failing the workload rc (the science is done
+ — exit 0 with a push-failed sentinel field instead).
+ - **Orchestrator side:** avoid pushing to the issue branch while lanes
+ are mid-run when feasible; when a mid-run push is required (a
+ crash-fix relaunch), expect in-flight lanes' terminal pushes to need
+ the fetch+rebase path above.
Scope / surfaces
- Primary target:
.claude/rules/pod-side-reporting.md. The enforcing code (the issue-driver push helpers) is experiment-side; the rule is the durable surface. Cross-check.claude/agents/experimenter.md§ launch contract for a one-line pointer if the planner deems it warranted.
Constraints / invariants
- Workflow-surface only.
scripts/workflow_lint.pyno-flags run passes; LESSONS.md row for pod-side-reporting.md already covers "pushing result commits" — update its fires-when text only if the planner deems necessary. - This session runs under
EPM_WORKFLOW_FIX_SESSION=1and carries aworkflow_fix_target:Provenance line — recursion guard applies.
Provenance
- workflow_fix_target: .claude/rules/pod-side-reporting.md
- fingerprint: ad5f3cc580c7
target_file: .claude/rules/pod-side-reporting.md bug_observed: The hallu lane finished all science (270 cells rc=0, Hub sidecars verified) then exited 1 at the final results-git push: the orchestrator's r10 fix commits advanced origin/issue-1739 past the instance's clone, the driver's push retry detected behind=1 but never fetched/rebased, and the healthy run was classified as a workload crash with poweroff. why_workflow_gap: pod-side-reporting.md owns result-push guidance but lacks a fetch+rebase-before-push requirement and an orchestrator-side mid-run-push warning; the race turns a completed run into a false workload crash. proposed_change: Add the fetch+rebase-before-push recipe (+ push-failure-without-workload-failure disposition) and the orchestrator-side race warning to pod-side-reporting.md. confidence: high related_task: #1739