EPS
← All docs·Activity

Daily — 2026-06-24 (clean-result v4 redesign shipped;

updated: 2026-06-25

What happened

  • Clean-result spec migrated to v4 (four-flat-H2: Takeaways / Goal / Methodology / Results + Repro/Context footer; methodology doc = mechanical copy of the body's ## Methodology). Shipped via workflow-fix merges + a v3→v4 prose-consistency sweep across analyzer/critic/codex-twin/research-project-structure. Dashboard gained an interactive per-point data viewer for v4 results (Phase 2/3, NaN-safe sidecars).
  • #658 (base-model foundation extraction) crashed deterministically at context #31 (f3_icl_json_k2, 30/50 marker JSONs) on both A100-80 and A100-40 — root cause an OOM from materializing full-sequence logits (152k vocab × seq) when the DV reads one slot. GCP's DELETE-on-exit destroyed all logs twice. Drove the GCP-workload-failure → RunPod failover + crash-diagnostics hardening (CLAUDE.md/compute-backend-failover.md; async-path wiring landed as #659, completed). #658 now on_hold under a user-directed 8×H100 RunPod re-implementation (slot-only logits + capture-completeness audit).
  • #653 (rank-ladder read/write) hit a per_pod_quota_exceeded (MooseFS ~130 GB) EDQUOT during select_checkpoint — merged_for_read/ dirs (~15 GB each) accumulated with no cleanup. Fixed across rounds 4–5 (strict-immediate-delete + merge inside try/finally + atomic completeness sentinel + 13 regression tests) plus a planner §9 "merge-disk budget vs per-pod quota" guardrail + critic Methodology lens 16. Now followups_running.
  • #661 (r_B extraction A/B/C) finished generation (raw completions safe on HF) then the Anthropic Batch judge wedged: a 90k-request single batch saturated the shared account, starving #661's 4800-request batch at 0/4800 for ~3h while an A100 idled. The deadline-less poll + no-resume client amplified it into respawn churn. Filed #662 (poll deadline/heartbeat/resume) and #663 (shard to ~8k/batch + resume); #661 blocked pending those.
  • Watcher false-positive respawns were the recurring theme: a healthy /issue 661 was killed mid-implementation because the stalled-detector keyed on a narrow 9-kind progress allowlist; fixed in-session to use the broad _latest_nonwatcher_event_ts (+ orphan/campaign siblings). Separately the idle-unmapped reaper was fixed to stop 35 abandoned detached-tmux sessions. #657 (sycophancy/activation-alignment) parked at awaiting_promotion.

Applied workflow improvements

  1. Target: scripts/task.py — what: post-marker/post-event --file - now reads the note body from stdin instead of crashing on Path("-").read_text() — commit: cc25f8acd8 Why: #658 orchestrator (session a0f08282, L253→255) ran task.py post-marker 658 epm:workflow-fix-candidate --file - <<'EOF' … and hit FileNotFoundError on the literal -; new-plan-version already supports --file -/stdin, so the inconsistency was the bug. Applied edit:

      note = args.note
  if args.file is not None:
-     note = Path(args.file).read_text()
+     # `--file -` reads the body from stdin (mirrors new-plan-version …)
+     note = sys.stdin.read() if args.file == "-" else Path(args.file).read_text()

    Verified: ast.parse OK, ruff clean, stdin branch returns the piped body in a simulated call.

  2. Target: .claude/rules/workflow-fix-on-bug.md — what: mandate that git push in the merge→push path is a standalone, rc-checked Bash call, never piped/chained so its exit code is masked — commit: 68dacad247 Why: #657 (session 09e911be, L179) — a piped merge+push one-liner masked a rejected push, so the documented pull --rebase=merges --autostash retry never fired; the orchestrator caught it only by manual inspection ("my pipe masked the failure so the retry didn't fire"). Applied edit:

      git -C "$REPO_ROOT" diff --cached --name-only   # must be empty post-merge
+ # PUSH-RC DISCIPLINE: `git push` MUST be a standalone Bash call whose exit
+ # code is checked — NEVER piped … (daily 2026-06-24 #657 masked-rejection)
  git -C "$REPO_ROOT" push origin main

    Verified: workflow_lint.py --check-references PASS.

  3. Target: .claude/skills/campaign/SKILL.md — what: add a "When NOT to use a campaign" routing check — a fully pre-specified phase DAG whose children share a sequential data dependency through a store routes as sequenced /issue, not a campaign — commit: 5dd89dceaf Why: #658 (session 0cc5cf53) — the orchestrator strongly recommended (and began spawning) a kind: campaign for a fixed 5-task extraction DAG sharing a base-model store; Thomas pushed back ("Wait are you sure we should use a campaign?") and the orchestrator reversed ("I over-reached recommending it"). The routing guidance didn't distinguish adaptive search from fixed DAGs. Applied edit: new ## When NOT to use a campaign (routing check) section before ## State surfaces. Verified: workflow_lint.py --check-references PASS.

  4. Target: scripts/task.py — what: accept --rationale as an alias for --summary on address-concern, matching defer-concern's flag name — commit: ae7ceeb2b9 Why: #658 (session a0f08282, L416→417) — address-concern … --rationale "…" failed with unrecognized arguments: --rationale because the flag is --summary while the sibling defer-concern uses --rationale; the inconsistency is a repeatable trip. Applied edit:

    - p.add_argument("--summary", default=None, help="…")
+ p.add_argument("--summary", "--rationale", dest="summary", default=None, help="… (--rationale is an accepted alias…)")

    Verified: ruff clean; address-concern --help shows --summary SUMMARY, --rationale SUMMARY.

Other problems & notes

Held — judgment calls (not auto-fixed):

  • Watcher respawns a healthy session during long Anthropic batch waits / GPU-idle (high — #658/#661). The stalled-detector was already broadened in-session (_latest_nonwatcher_event_ts), but a session legitimately quiet for hours waiting on an off-pod batch can still trip the self-report-freeze arm, and there is no GPU-idle advisory on the GCP poll path. HELD: editing scripts/autonomous_session_watch.py respawn predicates is high-blast-radius (fleet-wide crash-recovery) and was already touched 3× today; the cleaner root-cause fix is #663 (shard the batch so it doesn't wait hours). Suggested action: land #663, then if respawn-during-batch recurs, add a documented epm:waiting-on-batch exemption to the watcher's alive-but-stalled pass.
  • failure_classifier misclassified a deterministic repeat-crash as infra and the candidate was parked "for Thomas's morning" (high — #658). Round-1's crash was diagnosed infra and blind-respawned, wasting ~83 min + a second GCE instance; it was a deterministic code bug (same context on both runs). The fix (escalate to code when a respawn dies at the same phase/marker-count across attempts) needs cross-attempt state comparison I can't robustly verify tonight (verification gate). HELD; also flag the meta-lesson: a concrete in-scope candidate was parked, which violates the no-park rule. Suggested action: dispatch workflow-improver against scripts/failure_classifier.py (or file a small infra task).
  • task.py new does not persist goal:/kind: from --body-file frontmatter (med — #658). view 658 --json returned kind: None, goal: '' despite the body frontmatter, forcing a manual set-goal. HELD: ambiguous whether frontmatter should auto-persist vs the --goal/--kind flags being required-by-design (task-creation contract). Suggested action: confirm intended behavior, then either parse the frontmatter on new or warn loudly when a --body-file carries an unpersisted goal:/kind:.
  • configs/eval/default.yaml still defaults marker_token: "[ZLT]" (low — #658, noticed in passing). CLAUDE.md mandates ※ (id 83399), explicitly NOT [ZLT]. HELD under the science-meaning carve-out — flipping a marker default changes what marker experiments measure. Suggested action: file a small config task to set ※ and audit configs/leakage/* / configs/condition/* references.
  • GCP A100-40 rung accepts a workload whose prior peak GPU mem exceeded 40 GB (med — #658). The cost-ladder dropped a 54-GB-measured run to a 40-GB rung ("may OOM at 40 GB … let it ride"). HELD: needs a recorded per-attempt peak-mem to gate on, which may not exist; the immediate incident was masked by the code crash. Suggested action: if peak GPU mem is recorded per attempt, skip a rung whose capacity is below the prior peak in backends/gcp.py.
  • post-marker --by defaults to "unknown" (med — #658, 82 markers). The cleanest fix layer is ambiguous (a task.py default change is provenance-contract-adjacent vs editing many SKILL.md call sites) and impact is low. HELD. Suggested action: add --by orchestrator to the documented epm:consistency/epm:plan/epm:clarify/epm:progress post-marker invocations in issue/adversarial-planner SKILLs.
  • gcp.py crash-net (_eps_persist_diagnostics) partial-artifact capture + stale dispatch-handle on re-dispatch (high/low — #658). Both touch #658's own in-flight deliverable code (actively being reworked under the 8×H100 hold). HELD under the actively-changing-in-session defer rule. Suggested action: verify the crash-net uploads expected_artifacts.git_paths partials before DELETE, and invalidate .claude/cache/issue-<N>-handle.json on re-dispatch, once the #658 rework lands.
  • #657 sycophancy y-axis judged by Haiku, not Sonnet 4.5 (low). Reused #623/#612 base-rate data carries the pre-2026-06-23 Haiku pin. Task-state caveat, not a workflow gap (the standing rule is already correct). HELD judgment-call(re-judge vs caveat-and-ship). Suggested action: land the Haiku-judge provenance as an explicit caveat in #657's clean-result body, or re-judge the base rates under Sonnet.

Already fixed in-session (no action — recorded for completeness): GCP DELETE→RunPod failover + crash-diagnostics (#658→#659); slot-only-logits OOM fix (#658 rework); EDQUOT checkpoint-merge + planner §9 lens 16 (#653); stalled-detector broad-marker fix + idle detached-tmux reaper (#661 / watcher); poll_pipeline cpu_advancing running-max baseline (false multi-shard stall); #659 workload_cmd char-explosion bug + plan-v1 FAIL (caught by critic/reconciler/verify_plan); batch-judge hardening filed (#662/#663); v4 clean-result redesign + dashboard data viewer.

Transient / external (no fix — handled correctly): Codex plan-critic helpers crashed (app-server exited unexpectedly) on #658 and a dispatch-probe failure on #659/#661 — the documented Codex-no-show single-Claude fallback fired each time. Anthropic SDK batches.list() hung 2+ min from the VM (a raw curl returned in 0.23s) — prefer curl over the SDK for batch-health probes. Self-recovered one-offs: SendMessage called with prompt: before loading its schema; a 2-min Bash timeout on a heavy worktree checkout; an Overleaf main.tex modified-since-read race; a latest-marker --kind flag guess (flag is --prefix).

My thoughts

Highlighted results

  • no results promoted today