EPS
← All docs·Activity

Daily — 2026-07-15 (heaviest infra day on record — 35 fixes filed, ~15 shipped same-day; 0 promotions)

updated: 2026-09-03

What happened

  • Workflow-fix pipeline ran at record throughput: ~15 kind: infra tasks completed same-day (#1311–#1313, #1316–#1319, #1322, #1326–#1327, #1340, #1348, #1355–#1356, #1360, #1362–#1365 …), every code-review round single-Claude under the Codex org quota outage (sentinel live until ~2026-08-06; the #1204 pre-spawn skip worked everywhere).
  • Experiments: #1333 (marker implant reproduces the farther-but-not-more-diffuse full-fine-tune signature) reached awaiting_promotion at 23:58Z after a 7-attempt crash ladder; #1315 (impolite activation-shift geometry) passed upload-verification + terminated its pod; #1332 (directional function-space map predicts marker leakage) parked at awaiting_promotion; #1335/#1336/#1345 running; #1310's v3 prefill pivot crashed on the instruct arm but the base arm flipped the null (map positive) — parked on_hold; follow-up rounds ran on #825 (ridge-only Track-S), #952 (china-politics top-up), #928 (prefix-based mapping arms), #1090 (fu4 complete, fu5 started).
  • #1073 promoted useful (single greedy answer adequate vs 10-rollout-averaged targets) at 21:25 PT — promoted_at lands on 2026-07-16 UTC, so it appears in tomorrow's included_tasks, not today's.
  • Interactive results-synthesis day for Thomas: long #825/#1092/#1310/#823/#952 sessions produced Dan-facing writeups — and most of today's user-corrections (wrong R² range, in-distribution overclaim, byte-identical figure series; see notes).
  • Fleet pain: VM root disk / hit 100% at least 3 separate times (wedged commits, worktrees, Bash, the Step 9c gate across ≥6 sessions); ~12 spurious Usage-Policy refusal kills across 8+ sessions (recovery ladder held every time, ~1.7 h lost on #825 alone); crash-fix marathons on #1333 / #1345 / #1005 / #1335 all recovered autonomously.

Applied workflow improvements

Route-1 self-applies (2 — both user-memory files, outside the repo git, so no commit shas; verification = re-read after edit):

  1. Target: ~/.claude/projects/.../memory/feedback_browser_links_for_review.md — what: freshly-authored writeups also get the bare URL unprompted — commit: n/a (memory file, not in repo git) Why: three more URL asks in one day ("give me the URL" ffa3bb86 11:29Z; "output the URL" + "give me the url" 09f28ede) Applied edit: 
    + - **Freshly-AUTHORED artifacts count too** (writeups, result docs composed in-chat): publish the gist/blob/dashboard link and print the bare URL unprompted, BEFORE Thomas has to ask. Recurrence 2026-07-15: three more asks in one day.
  2. Target: ~/.claude/projects/.../memory/feedback_no_opaque_condition_codes.md — what: rule extended to dashboards + figure legends in ANY user-facing artifact — commit: n/a (memory file, not in repo git) Why: "what is stoch parent vs stoch fresh" (ffa3bb86 22:19Z) — opaque labels reached a dashboard Applied edit: 
    + **Also binds dashboards and figure legends in ANY user-facing artifact** — interim dashboards, quick figures, chat-shared plots, not only clean-results/plans. Incident 2026-07-15: "stoch parent"/"stoch fresh".

Route-2 filings (35, all via daily_drive_filings.py; bodies + ledger in logs/daily/filings-2026-07-15/; dispatch deferred to the watcher proposed_infra_sweep backstop by the 60s stagger). First 9 are the Step C parked-candidate routings (source task in parens); the rest are mined from today's transcripts, severity-ordered:

  1. filed for review #1380: codex-code-reviewer rubric gains Step 4.6 gate-scope check (from #1317)
  2. filed for review #1381: verify_plan flags false "in the no-flags default run" claims (from #1322)
  3. filed for review #1382: fix two pre-existing red workflow tests on main — test_living_docs registry-drift + followup-labels corpus replay, both re-confirmed red at filing (from #1326)
  4. filed for review #1383: verified-at-filing binding rule gains a context-consistency clause (the #1330-duplicate-of-#1309 incident) (from #1330)
  5. filed for review #1384: experiment-implementer per-arm-class smoke-coverage clause (from #1340)
  6. filed for review #1385: workflow_lint gains --check-rule-frontmatter-parses (from #1348)
  7. filed for review #1386: hub.py classifies Xet "maximum queue size reached" as transient (from #1359; wf_fix:false — src/ library)
  8. filed for review #1387: verify_plan flags §6 exit-0 criteria on pre-existing-red repo-wide lints (from #1365)
  9. filed for review #1388: fix the two bare-list_repo_tree step9c lint offenders (issue1073:102 + issue1092_inline_operator_stage:92) (from #1368/#1369; wf_fix:false — experiment scripts)
  10. filed for review #1389: verify-before-asserting gains compose-time re-grep + artifact-staleness legs — 6 sessions quoted wrong/stale numbers to Thomas today (HIGH; the R²-cosine range, the #779 overclaim, the #1310 stale-v2 re-cite ×3)
  11. filed for review #1390: interpretation-critic degenerate-series (byte-identical figure arrays) check — the #1092 8-series-really-2 figure Thomas caught (HIGH)
  12. filed for review #1391: watcher re-drives ACTIVE tasks whose session died with the registration file missing — the #1090 fu5 blind stall (HIGH)
  13. filed for review #1392: disk sub-floor sentinel triggers an immediate vm_disk_guard --apply (9.5 h of sidecar rows preceded today's 100%-full) (HIGH)
  14. filed for review #1393: inline/user-chat analysis rounds stage multi-GB HF downloads off / (the 14 GB #823 ENOSPC) (HIGH)
  15. filed for review #1394: planner reuse search covers LIVE sibling sessions' in-flight modules ("flag the consolidation") (HIGH)
  16. filed for review #1395: critic/verify_plan escalation on perm-battery wall-times not derived from the measured pilot (the #1092 2.6× undersize) (HIGH)
  17. filed for review #1396: extraction-store upload sequenced BEFORE long fit phases (the #825 stranded turnstore → GPU re-extraction) (HIGH)
  18. filed for review #1397: code-review throughput check requires naming the batched helper for per-cell fit loops (2 serial loops shipped past review today)
  19. filed for review #1398: first-pass neutral vocabulary for gate/kill-criteria briefs (refusal-kill prevention; ~12 kills today)
  20. filed for review #1399: wf-fix filing dedup also screens recently-COMPLETED siblings for topical overlap (#1350-vs-#1329, #1330-vs-#1309)
  21. filed for review #1400: triage-observer suppresses flags on an empty candidate window (2 false positives on #1005)
  22. filed for review #1401: pod.py sync env re-pushes + verifies GITHUB_TOKEN (pod-1333 ran attempts 5-7 on manual git-bundle sideloads)
  23. filed for review #1402: shared GCE staging gets the hardened HF-download retry wrapper (#1092 rf01 crash-loop)
  24. filed for review #1403: Step 10d late merge conflicts dispatched to a fresh subagent (the #1338 "Prompt is too long" death)
  25. filed for review #1404: PM spend-leak triage checks ownership before declaring an emergency + pod- prefix hazard note
  26. filed for review #1405: paper-plots interim-figure defaults (no hatching/n-labels/guess markers; CV fold captions; arm disclosure)
  27. filed for review #1406: SPEC.md — state the protocol delta when citing a sibling issue's headline (#779/#823 R² non-comparability)
  28. filed for review #1407: Monitor conditions on re-runs key on process exit / fresh mtime, not file existence
  29. filed for review #1408: Step 9c baseline compare runs in a detached scratch worktree (immune to shared-root dirt; #1317's 3.3 h serialization)
  30. filed for review #1409: smoke runs exercise data-dependent gates at smoke n (#1345's 4 never-executed gates)
  31. filed for review #1410: gotchas — /mnt/eps-data root is root-owned; stage under the per-issue worktree data dir
  32. filed for review #1411: gotchas — Edit tool un-escapes \uXXXX literals; write Unicode-sensitive literals via python heredoc
  33. filed for review #1412: Step 9b/10d binary-figure conflict resolution guidance (regenerable → take the newer round's copy)
  34. filed for review #1413: root-branch guard hook — fix residual ssh-argument-string false positive
  35. filed for review #1414: plan §9 pins the out-root FILESYSTEM + headroom probe (the #1333 wrong-mount ENOSPC)

Lint gate: skipped — zero route-1 edits touched repo workflow files this run (both self-applies were user-memory files outside the repo).

Other problems & notes

  • URGENT re-surface — #1038 (schedule the #681 worktrees→/mnt/eps-data cutover): / hit 100% three separate times today (≈19:31, 22:37–23:10, 22:52 UTC), wedging commits, worktree creation, Bash spawns, and the Step 9c gate across ≥6 sessions. #1392 (immediate reclaim) and #1376 (HF-cache tier, in review) are mitigations; the structural fix is the cutover sitting in proposed #1038 waiting for your quiet-window go-ahead. This is now the fleet's top recurring failure.
  • Codex org quota exhausted until ~2026-08-06 — every doubled review site fleet-wide ran single-Claude all day. The sentinel + no-show fallback worked as designed, but cross-family review diversity is off for ~3 weeks. Ops decision for you: wait for reset vs raise the org limit.
  • Review-stack latency skepticism (your words): "are the multiple code reviews actually helping?" + "why is it taking so long" (#825 spent ~2.5 h in plan+review before compute). Signal for prioritizing the workflow-v2 report-only pipeline dogfood — no filing (v2 exists; adoption is your call).
  • Agent-memory lesson not applied by /daily (agent-memory writes moved to the consolidate_lessons.py cron, #711): experiment-implementer memory should gain "report SHAs only from git rev-parse, never typed from memory" (#1345 r6 fabricated a SHA tail; mirrors the orchestrator-level never-fabricate-SHA memory). Left for the nightly consolidation cron; noted here so it isn't lost.
  • Skipped filings (already fixed / already covered), for the record: GCP GPU-util≈0 poll advisory — landed 2026-06-29 via #730 (backend_poll.py L433-505; residual question why it didn't fire on #1005's 4 h CPU run is a possible future filing); #1369/#1371 torch-before-dotenv offenders — verified fixed on current main (test passes); batch-03's "4 main-red scripts" claim — 2 fixed on tree, the live 2 are #1388; hub retry_transient lint (#1318), width re-eval (crash-fix-rounds.md), spawn_session unregister (#1327) — all grep-verified present.
  • Same-day self-healing (no /daily action needed): the fleet itself filed AND shipped ≈10 workflow fixes today — #1341 (root-draft step9c oracle), #1343, #1344, #1348, #1355, #1356 (zombie vLLM pid-namespace OOM), #1358, #1361 (pod git 403 gotcha), #1362, #1313 (judge 529 transport) — plus #1334 (pod --name-suffix) and #1326 (Step-0 guard inline-driver blindness).
  • Thomas corrections today (the substantive ones): wrong R² range in a saved writeup (0.60–0.93; the 0.93 was a cosine → corrected to 0.60–0.63); "map beats persona vectors" overclaim read off the in-distribution LOPO map (retracted; the standing CLAUDE.md verify-before-asserting pattern → enforcement legs filed as #1389); #1310 stale v2 summary re-cited 3× after v3 flipped the null; the #1092 byte-identical-series figure ("audit the per turn mapping result again because it doesn't make sense" → #1390); false $128/hr spend emergency on legit team pods ("no leave them" → #1404); "don't do PCA!" (#779 — code already complied); 5 figure-style fixes + "n=4998" fold-caption misread (→ #1405); "no i want: assistant in chat template vs …" re-spec (→ filed as #1345 same day); "run ridge only and diagnose why MLP failed" (#825); #922 scope narrowing to the token-to-token linear map.
  • Refusal-kill day: ≈12 spurious Usage-Policy kills across #1313/#1332/#1333/#1335/#1336/#1345/#1348/#825 — the (a)/(b)/(b2) ladder recovered all (one sonnet-pin respawn itself thrash-died and was micro-scoped per the documented recipe); prevention leg filed as #1398.
  • Transcript coverage: 88 transcripts had genuine 07-15 content (≈70 sessions; 10 parallel miners); ~20 mtime-bumped stale or 07-16-only transcripts excluded per the stale-transcript rule.
  • retraction re-check: 11 marker-mined parked candidates scanned across 11 source tasks, 0 retraction-skips; 3 items skipped instead as already-fixed-on-current-tree (verified by test run / grep + context read at filing time: #1369-item-1/#1371, m04/#730, batch-03 P10).
  • Title-sync drift sweep (Step D): title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-11.md for the verbatim rows).
  • Daily stub push: landed via sync_repo_root.py after one non-ff rejection (ahead=12 behind=1 at 06:32Z — normal concurrent-root traffic).

Living-docs drift

  • living_docs.py check FAILs with 199 findings — the same drift as the open re-synthesis proposal (≈75 dead legacy anchors, relates_to ⇄ evidence mismatches incl. #722/#763/#494–#501, ~54 unlinked completed clean-results, 7 stale spec-role-header State-trailer rows). Finding count and categories identical to 07-14/07-13; docs/open_questions.md unchanged since 2026-07-07. (Tonight's recipe-hash reads 1b37008540f0 vs the recorded 078285c14444 — a normalization artifact, not new drift; content-identical to yesterday's run.)
  • drift proposal already open (logged 2026-07-07, extending 06-29/07-04); not re-proposing. The standing proposal remains: re-synthesize open_questions.md (living-docs updater/backfill) — needs your ok; not auto-applied.
  • Parked-proposal re-consideration (Step B): the one un-reconciled living-docs rejection (#713) has had no context change (no 07-15-UTC promotions, open_questions.md untouched) — not re-surfaced; no followup-parked-redundant markers exist on any on_hold task — 0 revival candidates.

My thoughts

Highlighted results

  • no results promoted today (#1073 was promoted at 21:25 PT = 2026-07-16 UTC — it lands in tomorrow's brief)