EPS
← All docs·Activity

Daily — 2026-06-29 (huge autonomous infra/workflow day; 38 infra tasks completed, 9 route-2 fixes filed)

updated: 2026-06-30

What happened

  • Massive autonomous infra/workflow push: ~38 kind: infra tasks reached completed, including the structural disk fix #681 (dedicated GCP data disk + ext4 per-task quotas), #711 (deterministic nightly lesson-consolidation cron), #713 (folded /weekly's load-bearing checks into nightly /daily; retired the weekly cron), #727/#735/#745 (HF upload hardening — folder-commit, 5xx retry wrapper, hf_transfer), #741/#743 (GCP 7d max-run-duration + 8×A100/8×H100 machine types), #747 (cheap CPU pod lanes), #739 (always-on LESSONS.md index), and a long run of workflow-fix:/daily-fix: tasks (#710, #712, #714, #716, #718–#750).
  • Experiments parked at awaiting_promotion: #661, #664, #667, #685, #715, #722, #734 (none promoted yet → included_tasks empty). #658 is in followups_running, #697 in interpreting, #662 archived.
  • Active runs: #665, #666, #742, #744, #750 (+ the 9 route-2 fix sessions filed tonight, below).
  • In-session firefights handled and fixed: a 4.05 TB WandB quota blowup (full checkpoints uploaded as Artifacts → committed fix b4474042b7), the #664 ~$530/12h idle-8×H200-during-upload burn, and the #658/#722 139 GB-store / serial-CPU-LOCO disk+throughput incidents.

Applied workflow improvements

No route-1 self-applied fixes today. Every workflow-fixable problem the transcript sweep surfaced was a behavior/logic change (route 2) or already fixed in-session with a completed task — there was no trivial, no-behavior-change doc/string edit to self-apply. The genuinely-new, recurring gaps the per-session workflow-fix protocol missed were filed for independent review (route 2):

  1. filed for review #751: GCP→RunPod failover relaunch must sync the new pod into pods.conf, and pod.py config --update/--refresh-from-api must CREATE a missing entry from the live API — the documented recovery path fails for an absent pod. (recurring: pod-664/697/658 false-dead) — backend_poll.py + pod_config.py
  2. filed for review #752: GCP intent→machine map under-provisions a 7B forward with hidden-state capture (eval/debug→L4/g2-standard-4 OOM); honor the plan's A100-80 and size capture phases at plan time. (recurring: #666, #744) — backends/gcp.py + planner.md
  3. filed for review #753: add a PreToolUse/lint guard rejecting bare | python/| python3 consumer-side invocations, mirroring --check-heredoc-dotenv — the uv run python rule is stated but unenforced and was tripped ~41× across 4+ sessions (one exit-127). — workflow_lint.py + .claude/settings.json
  4. filed for review #754: Step 9c test-verdict should default to a touched-file-scoped pytest subset (no xdist → the whole ~5800-test suite stalls and is harness-killed in sparse worktrees). (recurring: #736, #665) — issue/SKILL.md
  5. filed for review #755: reconciler must persist EVERY upheld finding into the brief (it dropped 2 of 4 on #715) and quote the exact SPEC.md clause when overriding a structure-lens FAIL. — reconciler.md
  6. filed for review #756: analyzer scope guard — never spawn its own critics/Codex, never self-drive the review round or flip status (it did on #722). — analyzer.md
  7. filed for review #757: same-issue follow-up loop must re-read the latest superseding epm:followup-scope marker before the planner snapshots (it keyed on a stale entry-time scope on #658). — issue/SKILL.md
  8. filed for review #758: codex-critic prompt-composer must source any inlined numbers from the plan/artifacts under review (it inlined fabricated PP numbers on #722) and pin+flag the marker snapshot as possibly stale. — codex-critic.md
  9. filed for review #759: watcher must check the live-session registry (and treat an active-subagent session as alive) before respawning a driver — it spawned duplicate sessions on #664/#665/#740 and raced a mid-Phase-1.5 respawn on #739. — autonomous_session_watch.py

Other problems & notes

  • No route-3 judgment-call items — nothing today matched the scientific-meaning / destructive / money / external / genuinely-ambiguous carve-outs.
  • Already fixed in-session (dedup, no re-file): WandB weight-upload-to-Artifacts (committed b4474042b7, default off); GCP hung-but-RUNNING VM detection (#669); async GCP→RunPod failover (#659); registry audit --repair + 13 drifted tasks (#724); living_docs.py check degrade-on-bad-task (#725); GCP EXIT-trap phase=done on OOM (#750, running); GCP 24h→7d max-run-duration (#741); GCP FLEX_START create 300s exit-4 (#736); HF 10k-files-per-dir + tree-listing 504 docs (#748/#749); per-file→folder-commit upload (#727); persona-vectors recipe divergence (#728); analyzer figure .meta.json cross-check (#729); vLLM gotchas (#717/#746).
  • WandB cache residual (recommend route-2 follow-up, NOT filed tonight): the 4 TB quota driver (weight uploads) is fixed, but vm_disk_guard.py still has zero wandb references — the ~34 GB ~/.cache/wandb is unmanaged and contributed to the /-full fleet stall. A small vm_disk_guard wandb-cache reaping tier (sibling to the scheduled uv cache prune) is worth a follow-up; held off filing to bound tonight's batch at 9.
  • Out of /daily scope (→ /memory-sleep): the user-global memory reference_gcp_quota_filing.md claims an 8× H100 grant in eps-persona-gpu-jun2026, but session 911be962 found no H100 quota (A100-80 is the confirmed 8-GPU lane). Stale-fact correction belongs to memory consolidation, not a project workflow file.
  • Low/one-off (no action): task.py latest-marker/list-markers take --prefix not --kind (agents guessed --kind 14× — a --kind alias would help but is minor); address-concern --summary silently caps at 200 chars (document the cap); post-marker --note with backticks triggers shell substitution (use single-quotes/--file); SSH-MCP ssh_execute caps at 30 s ignoring inner timeout; task_progress.py FileNotFound TOCTOU on a status move during the summarize cron; VIRTUAL_ENV=introsp/.venv leaks into EPS cron/session env (cosmetic warning ×14); new_worktree.sh fails-and-recreates on a resumed autonomous /issue when the branch/worktree already exists (recovery worked). py-spy not installed on the VM (blocked a live CPU-hang probe). Each survived/self-corrected in-session; surfaced here for pattern visibility.

Living-docs drift

living_docs.py check exited non-zero (drift present):

  • 23 completed tasks with has_clean_result appear in no question's evidence: #408, #416, #432, #451, #458, #463, #469, #475, #493, #494, #496, #498, #501, #502, #506, #509, #511, #515, #522, #531, #536, #543, #549.
  • Question spec-role-header State trailer (dated 2026-06-02) is stale relative to linked results #464, #517, #528, #529, #533, #546, #556 (all promoted 2026-06-09→16).
  • Proposal: re-synthesize open_questions.md (run the living-docs updater/backfill) to reconcile the drift above — needs your ok; not auto-applied.

My thoughts

Highlighted results

  • no results promoted today