EPS
← All docs·Activity

Daily — 2026-08-01 (34 tasks completed; 2 route-1 fixes; 25 filed for review; 0 promoted)

updated: 2026-09-03

What happened

  • Heavy pipeline day: ~1,764 task-state commits. 28 tasks created — the 17 daily-fix filings from the prior nightly (#1962–#1978), 10 workflow-fix filings from in-flight sessions (#1980–#1989), and 1 new experiment (#1979). ~34 tasks reached completed (40 epm:merged commits). 0 promotions.
  • #1979 (per-prefix leakage-predictor race — the redo of #1900 at prefix grain after Thomas's "RESULTS THAT RAN WERE PER QUERY WHICH IS TERRIBLE" correction) was created, ran its full pipeline (6+ crash-fix rounds), and reached awaiting_promotion the same day. #1902, #1895, #1946 also cycled into/at awaiting_promotion; #1739 and #1979 ran followups_running rounds.
  • #1739 dominated compute: OOD + new-arm box sweeps with a rough infra day — 5/10 GCE boxes OOM-killed rc=137 in the pilot phase, 4 orphan RunPod fallback pods billing unlaunched, FLEX_START preemption churn, and ~3× wall-time overruns on unmeasured legs (details in the filed fixes below).
  • Fleet-wide ambient degradation, handled per protocol: Codex quota outage all day (sentinel live until 2026-08-06 → every ensemble review round ran single-Claude) and an org-wide OTPM 429 storm ~02:00Z that killed and cleanly respawned ~4 sessions' subagents.
  • Interactive threads: the big writeup/dictionary session (features/SAE + figure work), the KV-map probe thread (a "keys recognizable early" claim self-retracted after the shuffle null), #1345 fits relaunch, and the #1482 SAE dense-label bridge run.

Applied workflow improvements

  1. Target: .claude/rules/gotchas.md — what: grouped route-1 doc-sync pair from the Step C parked-candidate sweep — commit: 6cc1901f51 Why: (a) park on #1913 (implementer prose follow-up): the SLURM-lane entry still said the lane "silently rsyncs the repo-root tree on main" — stale after #1913 (merge c5c4e0922a) made ALL dispatches rsync a materialized committed-tree snapshot (live-tree only under EPS_SLURM_LIVE_TREE_RSYNC=1) with a post-sync --dry-run --itemize-changes verify. (b) park on #1919 (critic prose follow-up): run-time monitors load gotchas.md but not plan-compute-sizing.md, so the #1773 sampled-window GPU-utilization rule (≥10 readings / ≥60 s, never one nvidia-smi point sample) had no pointer where monitors look. Applied edit: 
    + **Update (#1913, merge `c5c4e0922a`):** the lead sentence's "silently rsyncs the repo-root tree on `main`" is now historical — `_resolve_rsync_source` materializes a COMMITTED-TREE snapshot via the `git_cloner` seam for ALL dispatches (…) and a post-sync `--dry-run --itemize-changes` re-run raises a typed `rsync_partial_tree` error BEFORE sbatch (…)
+ - **GPU-utilization / "GPU-bound" / idle claims in run-time monitoring need a SAMPLED window (≥10 readings over ≥60 s, reported mean + peak) — never one instantaneous `nvidia-smi` read.** (#1773: "GPU pinned at 90%" from ONE sample; mean 12.6%; an H100 billed ~7 h at ~87% idle.) Full rule: `.claude/rules/plan-compute-sizing.md` § GPU-utilization sampled-window.
  2. Target: .claude/skills/daily/SKILL.md — what: pin the bfs ISO -newermt form + never-suppress-stderr on transcript enumeration (consolidated entry C37) — commit: f1e98cd484 Why: this run's own transcript enumeration silently returned 0 three times (session 056d2981, 06:28Z): find on this VM is bfs, which rejects the GNU 'YYYY-MM-DD HH:MM:SS TZ' timestamp form, and 2>/dev/null swallowed the error — a less attentive pass could have shipped "no transcripts". Applied edit: 
    + Enumeration trap: `find` on this VM is bfs — `-newermt` accepts ONLY strict ISO-8601 timestamps (`-newermt '2026-08-01T00:00:00-07:00'`); the GNU-find form is an "Invalid timestamp" ERROR, and with stderr suppressed the pass silently returns 0 files — never suppress stderr on the enumeration call.
    Lint gate after all route-1 fixes: workflow_lint.py --check-references PASS, --check-asks PASS.
  3. filed for review #1990: sidecar canary tolerant of foreign concurrent denies (Step C park on #1876, fp 253371f2c59b — 36-min gate re-runs burned on a concurrent-append false-FAIL)
  4. filed for review #1991: blanket-add latch misses equivalent-blanket spellings ./, :/, * (Step C park on #1977, critic)
  5. filed for review #1992: flaky settle-sleep equality assert in test_inline_lint_gate.py (Step C park on #1977, implementer)
  6. filed for review #1993: crash-fix-rounds.md — shared-module propagation clause + stale-artifact disposition items (HF resume-skip shards, derived sentinels) (C2+C6 — the preflight EBADF race crashed #1979 then #1947 6h later while the fix sat round-local; #1902's scrambled shards nearly survived on HF via resume-skip)
  7. filed for review #1994: test_workflow_lint_phase_done_check.py hangs at interpreter exit >240s — join probe threads or register surcharge (Step C park on #1880, fp 7afc39759a27)
  8. filed for review #1995: carry-over gate channel-A _PATH_RE misses include-tree prefixes (Step C park on #1915, fp 7ddb0181b608)
  9. filed for review #1996: Step 6d.2 tick-source comments omit the quiet-wait Monitor path (Step C park on #1924, fp 6438c76ac16a)
  10. filed for review #1997: RunPod fallback pod lifecycle — orphan RUNNING pods on failed launches, terminal rung resumed a sibling's deliberately-STOPPED pod-1739, unasked fallback spend (C1)
  11. filed for review #1998: --min-ram-gb inert for GCP GPU machine selection — fail loud instead of silently provisioning an undersized box (C4 — two #1739 boxes OOM'd behind the claim it was "the OOM fix")
  12. filed for review #1999: HF fan-out pre-stage-once + FLEX→STANDARD escalation after first preemption on long serial legs (C10)
  13. filed for review #2000: sizing duty binds teammate-dispatched box sweeps — measured 1-cell pilot + RSS basis before multi-hour legs (C3 — #1739's ~3× overruns + 5 OOM boxes)
  14. filed for review #2001: planner §6 unit-of-analysis (measurement grain) field + statistics-critic lens item (C5 — the #1900 per-query vs per-prefix redo)
  15. filed for review #2002: smoke-architecture contract — resume-matrix + one-real-unit-into-production-out-root requirement (C7 — 11+ defects across #1947/#1979 concentrated in unsmoked resume/salvage paths)
  16. filed for review #2003: trigger-dense first-spawn pre-qualification by target class (C8 — 7+ refusal-killed spawns across ≥5 sessions; mitigation applied only after kills)
  17. filed for review #2004: CLAUDE.md teammate bullet — named-teammate final text does not route to the lead; SendMessage string message requires summary; content probe before a deliverable counts delivered (C9+C17)
  18. filed for review #2005: Step 9c full-suite gate launches DETACHED on the first attempt + end-the-turn after launch (C11 — a bg wrapper harness-killed at 95% re-ran a ~40-min gate; 3 blocking TaskOutputs held a turn ~31 min)
  19. filed for review #2006: hand-rolled box scripts must carry the teardown/poweroff leg (C18 — done-but-billing GCE boxes ~1h idle)
  20. filed for review #2007: git gc wedge — reviewed git prune + worktree gc.log tier in worktree_audit.py (C20 — live probe: root gc.log + 29 worktree gc.logs, 9,359 loose objects)
  21. filed for review #2008: pooling-convention (span-mean vs prompt-final-token) named per vector in capture plans + glossary row (C22)
  22. filed for review #2009: bare-python-pipe guard false-positives on remote python3 -c inside ssh-quoted commands (C34; real guard located at .claude/settings.json inline PreToolUse, target corrected per call-hop duty)
  23. filed for review #2010: corpus-coupled gate test test_corpus_replay_all_historical_markers goes red on stale branches when new-form markers land fleet-wide (C12 — ~70 min burned on #1917)
  24. filed for review #2011: provisioner broken-host avoidance — pod-1947-b re-rolled the same dead host 3× (C13; construction site pod_lifecycle.py added to target per call-hop duty)
  25. filed for review #2012: gotchas + code-reviewer rubric — a finally that can raise must chain/suppress-and-log, never replace the in-flight exception (C14 — masked EADDRINUSE cost 2 relaunch rounds on #1947)
  26. filed for review #2013: guard_root_code_commit block message appends "DO NOT report this commit as landed" (C15 — a blocked commit was reported to Thomas as "committed, pushed" with a <sha>-placeholder URL)
  27. filed for review #2014: /issue SKILL Monitor until-loop guidance — count-DECREASE keying, never pgrep -c || echo in a condition, poll_pipeline for multi-hour sets (C33)

Dispatch note: the shared 5-session infra cap was FULL for most of the run — the majority of the filed tasks sit at proposed for the watcher's proposed_infra_sweep backstop (~10-min cadence) rather than having live spawned sessions (per-task dispatch state in logs/daily/filings-2026-08-01/filed.jsonl).

Other problems & notes

  • Verify-before-asserting cluster (C16) — the day's most concentrated discipline failure: in the writeup session (0ac15c23, 03:04–03:10Z) four artifact-coverage claims in a row were answered from memory and each refuted by a probe run only after Thomas's pushback ("my last two answers were both wrong in the same direction"), plus a "#1934's recovery did run" walk-back at 21:00Z and a #1739 "zero committed OOD-wide results" claim flipped by the very next probe. Existing CLAUDE.md rules cover this; recurrence logged as a pattern datapoint — under rapid Q&A the compose-time re-grep must fire BEFORE the first answer, not after pushback.
  • Route 3 / needs-human: the rescue-stash triage item (C36) is already tracked in #1922 (shared: 319c2bf16e7c,kept,rescue,root-sync,stash,triage) — duplicate filing suppressed (#1483); freshness epm:progress note posted on #1922. Still live at this run's own sync: stash@{0} (319c2bf16e7c) + the 243e519e09a6 rescue patch.
  • Step C sweep hygiene: 9 parked candidates enumerated, all dispositioned — 6 filed (#1990 #1991 #1992 #1994 #1995 #1996 with routed-records posted on #1876/#1880/#1915/#1924/#1977), 2 route-1 self-applied (parks on #1913 + #1919 → commit 6cc1901f51, routed-records posted), 1 deferred again (the #1771 architectural-greenlight mechanization park — #1771 itself is still plan_pending rewriting that exact gate; re-evaluate after it merges). 1 benign skipped row (#1333 json-decode, relevant_kind: false, known shape); unmatched_record_fps ~300 refs — the standing #1703 advisory backlog, unchanged in character.
  • Retraction re-check: 7 marker-mined parks scanned (fresh events-tail read per park), 0 retracted upstream; 1 consolidated item suppressed by the freshness gate — the "dropped bqladder rerun" (C19) is in fact tracked on #1739 (BQLADDER HANDOFF — deferred to GCP and re-owned there, epm:progress 2026-08-01T22:46Z) — skipped filing.
  • Refuted at compose time (not filed): C38 (detached pidfile idiom) — .claude/skills/issue/SKILL.md § Detached VM-side phases ALREADY pins the exact bash -c wrapper/$! idiom; the #1947 pidfile incident was non-application of the existing recipe. C35 (cert TTL re-lint) — certs are already content-hash-bound; stale = matching sha past MAX_AGE is the documented design, so the redundant re-lint is the intended safety margin.
  • Already tracked / landed, no filing: carry-over bare-name over-FAILs (C23) → open #1982 (both incidents deduped in-session); #1895 branch-sync 20s-cap rc=124 (C24) → #1981 (filed in-session); #1979 judge_raw upload gap (C25) — HF-probed closed (15 files present under issue1979_prefixrace/judge_raw); preflight EBADF code fix landed (22c2ddb2d3), #1739 RSS guard landed (eb06ab7e34) — only the workflow residues were filed (#1993, #2000).
  • Note-only recurrences (existing rules, mechanical guards did their jobs): Codex quota outage all day (C27, watcher codex-outage pass owns alerting); OTPM 429 storm ~02:00Z, ≥4 sessions, all clean respawns (C28); piped-git guard blocks ×3 (C29); Edit/Write-before-Read slips ×7 across 7 sessions (C30); inline/background split typed by Thomas ×3 — the promoted 2026-07-29 ack-proposes-the-split rule failed those turns (C31); interactive scope/register re-steers incl. "use ALL the features" ×2 and "less jargon" (C32); grouped one-off tool/shell slips + guards-worked near-misses (C39); #1892 wf-fix filed on a refuted premise — the landed-fix compose check exists, enforcement miss by the (out-of-window) filer (C40); bg-task lifecycle noise (C41); a KV-map "keys recognizable early" claim shipped pre-null then self-retracted (C42); handled-by-design infra events — MooseFS stale-read crash-fix, GCP queue-vanish→RunPod failover, upload-verifier gap-close, #1945 CONFLICTING-PR recovery, #1947 GPU-6 hardware quarantine (C43).
  • #1345 fits relaunch recomputed the full ~84-cell lattice with no resume-skip on a billed bigmem box, and its first ETA was a guessed basis revised ~4× in 12 min (C21) — run finished; both halves covered by existing rules/finished script.
  • Run-internal frictions this /daily run: 2 concurrent-commit collisions at the shared root (a transient HEAD.lock, then a cannot lock ref 'HEAD' race) — both absorbed by the bounded retry; the driver's sha-scanner WARNed on 5 hex tokens in the c8 body that are session basenames (correctly cited as sessions, not commits; no action).
  • retraction re-check: 7 marker-mined problems scanned, 0 skipped for retraction; 1 (C19) skipped as already-tracked-upstream.

Living-docs drift

  • living_docs.py check exited 1 with 199 findings — byte-identical drift hash 078285c14444 to the 07-07/07-22…07-28 rows (relates_to ⇄ evidence mismatches, completed clean-results missing from question evidence, stale spec-role-header State trailer, etc.).
  • drift proposal already open (logged 2026-06-29/07-04/07-06/07-07 lineage; hash unchanged); not re-proposing. Event-stream row appended for tonight.
  • Step B: the sole un-reconciled living-docs rejection (#713) is unchanged with no new context — not re-surfaced (no-spam skip); no on_hold parked-redundant follow-ups exist.
  • title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-25.md). Read-only sweep exited 0.

My thoughts

Highlighted results

  • no results promoted today