EPS
← All docs·Activity

Daily — 2026-06-11 (19 tasks filed, 15 results parked, 18 follow-up rounds; 2 workflow fixes auto-applied)

updated: 2026-06-12

What happened

  • Highest-throughput day on record for the pipeline. 19 tasks filed (#595–#613), 15 experiment results reached awaiting_promotion for the first time (#491, #556, #561, #570, #571, #585, #591, #594, #600, #601, #602, #603, #604, #605, #611), and 3 infra tasks completed (#587 ETA display, #588 GCP workload-cmd lanes, #596 sparse worktrees). #593 archived as superseded by #591; #499 archived.
  • The same-issue follow-up loop was the workhorse: 18 rounds completed in one day (ft-content-control on #491, seed2 replication on #537, fractional-epoch-r16 on #546, contrastive-2x2 round-3 on #552, exposure-matched-ckpt300 on #561, saturated-install-em-eraser on #570, persona-split-composition on #571, step6to12 sweep on #585, e5-em-rejudge on #591, probe-genre on #594, svd-titration on #597, geometry-plus-prior on #541, posonly-multiepoch on #601, shuffled-replay-l27 on #602, base-text-extraction on #603, post-response-slot-key on #604, wider-marker-panel on #605, syco-best-geometry r4 on #480). More in flight at write time (#597 dense-early-contrastive-grid, #608 sub-ceiling-install, #533 bare-word grid).
  • First full day with GCP as the exercised primary lane, and it surfaced a cluster of new failure classes — GCE metadata-runner killing workloads on long progress-bar lines (bufio.Scanner, #491), boot-disk default too small (#606), sibling sessions burning each other's A100 quota attempts (#608/#597), detached-workload sentinel gap (#601), preflight-JSON parse crash on GCP boots (#602 ×3) — every one of which got an in-flight workflow fix + gotchas entry the same day.
  • The workflow-fix-on-bug machinery had its biggest day: ~50 epm:workflow-fix-applied markers across tasks (GCP quota pre-check + stderr surfacing, boot-disk guard, verify_uploads reproducibility-card merge, codex_task.py output-clobber fix, runpod_api SUPPLY_CONSTRAINT classification + spot-lever disable, vm-disk auto-remediation, worktree_audit flock, poll_pipeline cwd-anchoring, and more). 16 epm:failure-lesson markers posted; all generalizes: yes lessons were persisted to agent memories / gotchas in-flight.
  • Still running at write time: #522, #542 (negative-distance dose), #545 (K1 gate iterating, see notes), #597, #599 + #606 (interpreting), #608, #612.
  • No promotions during PT June 11 — #509 and #549 were promoted late June 10 PT and already counted in yesterday's daily.

Applied workflow improvements

  1. Target: .claude/agent-memory/experiment-implementer/ — what: failure-lesson consolidation: merged two duplicate memory entries for the same #602 preflight --json lesson into one — commit: 5ab22a693 Why: #602 hit the identical parse bug twice in one day (pod launch 07:24Z, then 3 GCP boots of the follow-up round at 03:33Z next day); each epm:failure-lesson persisted its own memory file (feedback_preflight_json_multiline.md + feedback_preflight_json_parse.md), giving the agent two index lines for one lesson. Applied edit:

    - .claude/agent-memory/experiment-implementer/feedback_preflight_json_multiline.md (deleted; content folded into feedback_preflight_json_parse.md)
- MEMORY.md: - [preflight --json is multi-line](/docs/feedback_preflight_json_multiline) — ...
+ feedback_preflight_json_parse.md **Why:** now records BOTH incidents + the --skip-preflight escape-hatch advice from the deleted twin

  2. Target: .claude/rules/gotchas.md — what: new entry: stale VIRTUAL_ENV (e.g. ~/introsp/.venv) uv mismatch warning is ignorable noise — commit: 4c66faaea Why: 381 occurrences of warning: VIRTUAL_ENV=/home/thomasjiralerspong/introsp/.venv does not match the project environment path .venv across today's session transcripts; two problem-sweep agents flagged it as a suspected environment fault, and it visually adjoins unrelated crashes. Documented nowhere until now. Applied edit:

    + - **Stale `VIRTUAL_ENV` from another project (e.g. `~/introsp/.venv`) makes every `uv run` print a mismatch warning — it is noise, not a fault.** ... do NOT chase the warning ... run `env -u VIRTUAL_ENV uv run ...` where that matters.

Other problems & notes

  • #545 cost watch: pod-545 (8×H100 RunPod) has been alive >24h through 5 failed P1 attempts (21/25 → 8/25 → 5/25 cell failures) and 2 K1-gate FAILs (round 2 at 06:28Z: components 1+2 now PASS, third still failing). The session is actively iterating a denominator fix, but this is the most expensive open loop of the day — worth an eyeball on whether the K1 gate design needs a re-plan rather than more relaunch rounds.
  • #537 burned ~47 min on 12 consecutive HF adapter download incomplete AssertionErrors in phase-2 generation (07:06–07:53Z) before recovering. Suggested: adapter snapshot downloads need retry-with-backoff / list_repo_files verification at the fetch site (experiment-code fix via experiment-implementer; the snapshot_download truncation memory exists but the fetch path didn't use it).
  • GCP A100 quota contention between sibling sessions (#608 burned attempts 4–5 on QUOTA_EXCEEDED while #597's follow-up sat pod-pending; 8/8 GPUs in us-central1 were already held by sibling eps-issue-* VMs). The in-flight fix (quota pre-check + stderr surfacing, commit 22bb9848a) stops the attempt-burn; residual gap = no cross-session quota arbitration/queue. If contention recurs, consider a VM-side quota lease (scripts change — your call).
  • task.py raise-concern --summary 200-char ValueError hit in ≥2 sessions (5+ consecutive retries in #545's session; once in a main-dir session at 246 chars). The cap is already documented in code-reviewer.md and twice in issue/SKILL.md, so no doc fix applied; the residual fix is in scripts/task.py itself (auto-truncate to 200 with detail shifted to --evidence, or print the limit in argparse help) — orchestration-code change, your call.
  • Foreground sleep N && <cmd> chains were hook-blocked in ≥6 sessions (waiting on codex output files, ssh kill -0 checks, tmp logs). Every block recovered via Monitor/bg-Bash; rule already lives in CLAUDE.md + SKILL.md, so no new doc — flagging the frequency only.
  • Edit-before-Read tool-contract violations across many sessions (8+ retries in #521's session alone) — pure retry waste; harness-level contract, no workflow-file fix available.
  • Codex companion flakiness continues: stall force-cancel at >600s, No job found probe errors across 3 sessions, and an output-file no-show on #491's clean-result round 1 (posted epm:failure, fell back Claude-only per protocol). 5 epm:codex-task-failed markers today vs ~150 completed. In-flight handling worked; no action beyond continued monitoring.
  • Worktree-lock collisions: cleanup attempts on locked agent worktrees failed with cannot remove a locked working tree in the #533/#521/#537 sessions. All were LIVE holders (the lock did its job); the unlock-first recipe for done agents already lives in workflow-fix-on-bug.md. No fix — noise, but verify the worktree sweeper isn't retry-looping on live holders.
  • /tmp handoff fragility: #585's session wrote a body file to /tmp via bash redirect and task.py later got FileNotFoundError on it. Suggested action: prefer tasks/<status>/<N>/artifacts/ for cross-step handoffs that must survive the session.
  • One session left a merge conflict in scripts/backend_poll.py after a plain git pull with unstaged changes (the documented recipe is --rebase --autostash); resolved in-session, reminder only.
  • #599 sparse-checkout pathspec friction ("paths matched outside your sparse-checkout definition") — side effect of the new sparse-worktree default from #596; recovered in-session. Watch for recurrence before changing the default.
  • Tool-parameter mistakes caught by harness validation (no retries beyond one each): change_title with undefined title (#585), TodoWrite string-vs-array (#491), TaskOutput timeout type, ScheduleWakeup missing prompt (auto session).
  • Experiment-code one-offs, each handled in-session: fig.set_title AttributeError + savefig_paper kwarg drift in #480 plotting; ModuleNotFoundError: i509_syco_conditions; ruff missing in #522's venv; python-pptx missing + Google Slides API not enabled in the slides session; preprobe_ladder.json missing in #533; assorted KeyError/TypeError data-shape crashes (#591 role-field, #406-gram key).
  • Documented anti-patterns violated once each (rules exist, no new fix): a hand-built tasks/completed/"$n" path with broken quoting crashed an inline loop (use task.py find); #568's worktree bootstrap used heredoc find_dotenv() (AssertionError) instead of the canonical set -a && source .env, plus HF_TOKEN missing at spawn.
  • rsync protocol errors (code 12) during pod syncs in #533 + auto sessions, and a stale-tmux-socket + 42 GB uv-cache timeout episode (already auto-remediated by the new vm-disk watcher pass from #587). Transient infra.
  • Repo hygiene: git is warning too many unreachable loose objects; run 'git prune' (.git/gc.log present) on the shared root. Safe to run when convenient, but it races with concurrent sessions — suggest a quiet moment: rm .git/gc.log && git prune.
  • Sweep-coverage caveat: one of the 7 transcript-sweep batches (19 transcripts covering #537/#541/#546/#561-adjacent sessions) reported zero problems; its sessions' only user messages were normal steering ("Actually, just remake figure 2…"), not corrections. Possible under-detection in that batch.

My thoughts

Highlighted results

  • no results promoted today