workflow-fix: intra-phase checkpointing for >1h serial loops
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by the workflow-fix-on-bug protocol from a prose follow-up surfaced by the #823 phase-4 optimization audit (emitting agent: read-only optimization-audit subagent; routed by the orchestrator).
Goal
Extend the checkpoint-per-phase code-style rule to require intra-phase checkpointing for serial fit/solve loops exceeding ~1h projected wall-time, and add the corresponding code-reviewer check.
Workflow gap
- Bug observed: #823 phase 4 accumulated ~20h (unpatched) / ~3.7h (patched) of serial fit results purely in memory with a single terminal JSON write (
ridge_r2_by_arm.json, run_823.py:1704-1706), so every restart/crash forfeits all completed fits; this directly blocked a user-directed restart-with-optimization (the restart would have re-run ~2h of already-finished work, flipping the decision to RIDE). - Why it is a workflow gap:
.claude/rules/code-style.md's checkpoint-per-phase rule stops at PHASE grain — a multi-hour serial loop INSIDE one phase satisfies it while being fully restart-hostile; no code-reviewer rubric item asks whether a long serial loop persists partial results, and five review rounds on #823 did not flag it. The same gap amplified both #823 GCE crashes' cost (any phase-4 progress at crash time was unrecoverable). - Confidence (emitter): high.
Proposed change (candidate diff sketch — refine in planning)
.claude/rules/code-style.md (checkpoint-per-phase section):
+ Intra-phase checkpointing: any serial fit/solve/eval loop whose projected wall-time exceeds ~1h
+ (count x per-call cost, per the plan §9 sizing) MUST persist per-cell (or per-arm/per-layer) partial
+ results as it goes — atomic append (JSONL) or per-cell files + a .done sentinel — and skip completed
+ cells on restart. In-memory accumulation with a single terminal write makes every crash/restart
+ forfeit the whole loop (#823 phase 4: ~20h of fits, one JSON at the end).
.claude/agents/code-reviewer.md (rubric):
+ Long-loop restartability: for any serial loop the plan sizes at >1h, verify partial-result
+ persistence + a resume predicate exist; absence is a substantive finding.
Scope / surfaces
- Primary targets:
.claude/rules/code-style.md,.claude/agents/code-reviewer.md - Related (planner side) is already covered by #869's Fix C (serial-fit-loop sizing); keep the two consistent — this task adds the RESTARTABILITY requirement, #869 adds the SIZING requirement.
Constraints / invariants
- Workflow-surface only — never experiment code,
configs/, ortasks/. scripts/workflow_lint.py --check-askspasses; ruff on touched files passes.- This session runs under
EPM_WORKFLOW_FIX_SESSION=1and carries aworkflow_fix_target:Provenance line — it MUST NOT auto-route any of its own subagents' workflow-fix candidates (recursion guard).
Provenance
- workflow_fix_target: .claude/rules/code-style.md, .claude/agents/code-reviewer.md
- fingerprint: e33c73c22d90
Evidence (task #823, 2026-07-02): optimization-audit subagent measurements at 18:33Z — phase 4 run_823.py::phase4_ridge_refit (worktree issue-823, lines 1449-1708) accumulates r2_refit / per_ctx_r2 / r2_transfer_vals in memory, single write at lines 1704-1706; restart-vs-ride decision flipped to RIDE solely because a restart forfeits unpersisted fits. Distinct from #869 (fingerprint 965c890415f9 — sizing/trigger-broadening); this task is the restartability half.