EPS
← All docs·Activity

Daily — 2026-07-23 (915 commits; 2 clean-results parked; ~26 wf-fixes landed; 21 filed tonight)

updated: 2026-09-03

What happened

  • 915 commits landed on main. #1586 (four fiction-character leakage-lattice run) dominated with 176 commits: 4×H100 RunPod run after a ~5.5h GCP 8×A100 queue-starve, ≥7 overnight crash-fix rounds (disjointness assert, vLLM EngineCore deaths, ENOSPC from ~456 GB rung accumulation, HF LFS 403), pooled H1/H2 lattices + 12 paper figures landed. Cost ~395vsthe395 vs the 200–300 approved estimate (disclosed; crash rounds + 3.7× wall-time deviation).
  • Two new clean-results parked at awaiting_promotion same-day: #1615 (the persona vector's pre-image under the context→answer map is interpretable — its top activating contexts make sense) and #1639 (the four fiction-character context→dialogue maps are one dominantly shared linear operator — pooled map).
  • The workflow-fix pipeline landed ~26 fixes (#1614–#1642 all completed), incl. the research-pm.md ratchet trim (#1618), root-commit foreign-staged-file handling (#1620), inline both-mapping-arms + PNG-check duty (#1625), Step 9c/10d gate single-flight guard (#1627), /daily own-paths commits (#1630), and the pid-file-from-launch-chain contract (#1634).
  • Compute friction day: GCP queue-starve → user forced RunPod on #1586 and #1112; the RunPod fallback was FALSELY blocked by the shared-account hourly cap counting the fellows fleet's ~$2,855/hr (filed #1652); fresh pods refused SSH after fellows onboarding dropped the VM key from the team account list (filed #1655).
  • Codex quota exhausted until 2026-08-06 — every ensemble review site ran single-Claude per the #1204 sentinel (mechanism worked as designed).

Applied workflow improvements

  1. Target: .claude/rules/gotchas.md — what: new entry — benign httpx RuntimeError: Event loop is closed (AsyncClient teardown) matches naive crash-grep patterns and fires false crash probes in monitor loops; exclude the signature / key crash detection on exit code — commit: 42bfb88633 Why: #1112 monitor fired a false crash probe at 07:39Z on 2026-07-23 and was relaunched "benign-httpx filtered" (transcript sweep, batch 3). Verification: lint gate workflow_lint.py --check-references PASS after commit. Applied edit: 
    + - **Benign `RuntimeError: Event loop is closed` (httpx `AsyncClient` teardown) matches naive crash-grep patterns and fires FALSE crash probes in monitor loops.** ... RULE: exclude this known-benign signature from crash-signature greps (or key crash detection on process EXIT CODE + a traceback whose deepest frame is workload code) ...
  2. filed for review #1643: runpod_api.py:488 UP037 reds test_ruff_policy on pristine main (4 parked candidates: #1618/#1624/#1629/#1638).
  3. filed for review #1644: 3 issue1586_*.py entrypoints import torch before load_dotenv, redding the thread-caps invariant fleet-wide (wf_fix: false — experiment code; dispatch deferred to watcher backstop).
  4. filed for review #1645: register 3 unregistered Step 9c pin tests (ownership-probe + 2 daily-skill pins; #1546 class).
  5. filed for review #1646: Step 9c map-mode slow-file timeout surcharges + stale test_workflow_lint wall figures (measured 1188s vs 900s surcharge; #1634's gate was timeout-killed at 300s).
  6. filed for review #1647: 9a-ter inline lint gate single-flight hook (#1606 pattern, distinct site; unblocked by #1627 landing).
  7. filed for review #1648: workflow_lint check for fenced bare git commit -m recipes lacking a pathspec (7dbde267f1 class).
  8. filed for review #1649: step9c_baseline scan-set dirt wedge — exit-2 INDETERMINATE on any third-party untracked root draft (low-confidence park from #1632; filed per standing directive).
  9. filed for review #1650: poll_pipeline pid-identity check + reconcile the experimenter pgrep memory with clause 1d (#1634 deferred both).
  10. filed for review #1651: implementer pin-sweep report states its sweep universe (sweep_scope: token).
  11. filed for review #1652: scope the RunPod account-hourly-cap guard to EPS-managed pods (fellows fleet ~$2,855/hr falsely blocks EPS provisions; user policy correction 2026-07-23).
  12. filed for review #1653: completed-unmerged watcher respawn arm (refusal-killed mid-Step-10d leaves merges stranded — #1540 16h, #1622 ~2.3h today).
  13. filed for review #1654: HF upload-quota preflight at representative scale (2MB probes passed while production checkpoint uploads 403'd on billing state; dispatch deferred to watcher backstop).
  14. filed for review #1655: provision preflight verifies the VM SSH key is in the team account key list (fellows onboarding dropped it; 5 publickey-denied firings).
  15. filed for review #1656: detached runs declare a session-independent harvest step at launch.
  16. filed for review #1657: Step 10d updates the head branch before the first merge attempt (2 same-day "head out of date" first-attempt failures).
  17. filed for review #1658: interim-writeup conventions — plots not tables, novel results only, plain-English gloss, one-color-one-meaning (≥4 sessions of repeated user corrections).
  18. filed for review #1659: inline/teammate rounds size timeout fences from a measured 1-cell pilot (a self-set 3000s fence killed a healthy ~25 min/cell run; estimates off ~10×). Distinct from held #1635 (scientific-discipline half).
  19. filed for review #1660: ddp_timeout default for ZeRO-3 multi-GPU runs (wf_fix: false — training code; #1112 arm B NCCL watchdog crash at step 40/750; dispatch deferred to watcher backstop).
  20. filed for review #1661: agent-spec-size ratchet check in pre-commit (the research-pm breach entered via a direct-to-main commit no gate covers).
  21. filed for review #1662: suffixed follow-up pods terminate unconditionally on verified-done (pod-1586-b idled at ~$12–13/hr behind an ask-gate).

Other problems & notes

  • Route 3 (needs you), filed #1663 — spends-money carve-out: should the router auto-escalate to RunPod after a bounded GCP queue-wait on user-prioritized runs? Twice today (#1586 ~5.5h, #1112) you ended the wait manually with "just run on runpod". Suggested action: if wanted, define the trigger (tag + hours) and greenlight; filed --no-dispatch, tagged needs-human.
  • already tracked in #1636 (shared: root/commit/hygiene): bare repo-root git commit PreToolUse hook (parked on #1630 as architectural — a fleet-wide hook changing every session's commit ergonomics). Freshness note posted on #1636.
  • retraction re-check: 21 marker-mined parked candidates scanned, 5 skipped — research-pm ratchet ×2 (landed via #1618, commit 8ede3ddb1b; current size 46,785 B < 47,000), issue1092 dotenv (landed via #1619, commit ab75ff5b05), fixture-denominator tests ×2 (#1633/#1634 parks — both named tests PASS on current main, verified tonight). Every filed candidate was re-verified against the current tree at compose time (pytest/grep evidence in each body).
  • Refusal-wedge incidents (2): the #1621 session lost ~46 min to 3 consecutive Usage-Policy kills right after a guard-surface implementer report (watcher respawn recovered); the #1622 session was refusal-killed mid-Step-10d leaving PR #1399 stranded ~2.3h (watcher flagged; manual recovery — the respawn-arm gap is filed as #1653).
  • User caught a train-on-eval-overlap fairness gap in the #1092 persona-corpus comparison; a battery-held-out fair recompare was run after the fact. Same inline-round discipline theme as held #1635 (which also gained today's incidents: missing null challenged in #779's writeup was yesterday's #1635 incident 2; today's siblings folded there conceptually — no new filing).
  • VM contention: shared-VM kills twice in one session (cert run exit-144; #1310 instruct arm OOM-killed ~2.5h redone). No single concrete fix named — choom/thread-caps exist; cross-session load coordination between Step 9c gates and long batteries remains an open design question (not filed; too vague to dispatch).
  • Concurrency noise, all auto-recovered by existing recipes: ~4 .git/index.lock collisions, ~4 push rejects → sync_repo_root recoveries, 2 Step 10d first-attempt merge failures (fix filed #1657), one wedged rebase autostash-recovered. Guard hooks correctly blocked piped git pushes (×4 sessions), bare | python -c, uncertified root code commits (×2), and a repo-root git checkout --detach attempt.
  • Adherence slips (no new rule needed, rules exist): one fabricated-SHA-in-URL self-caught by the 404 check; "where is the URL" fired once BEFORE #1622's fix landed same day; user polled "is this done?" 3× over an hour after a long inline round (completion-delivery discipline); user stopped 4 background agents mid-session (over-spawning signal); one "did you write this up into a clean result" re-ask 21 min later.
  • Writeup clarity friction: ≥5 "what does this mean" decode requests + one falsified over-compressed claim ("uniform 0.6 shrinkage") + mentor-writeup imprecision caught by user — conventions filed as #1658; jargon-density feedback also matches the standing explain-intuitively memory.
  • fp32 CPU 7B load OOM (earlyoom exit 143/144, 3 firings) recovered via bf16 downgrade — compute-placement judgment, one-off.
  • Step B (parked-proposal re-consideration): one un-reconciled living-docs rejection (#713, June); no new results bear on it — not re-surfaced. Zero parked-redundant follow-ups on hold — nothing to revive.
  • Title-sync drift sweep (Step D): title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-11.md for the verbatim rows).
  • staged-set audit: clean on both commits tonight (only this run's own files staged).
  • One dropped observation for the record: dashboards over-redacted LMSYS rows (user asked twice for actual text) — already tracked in held #1637.

Living-docs drift

  • living_docs.py check: FAIL — 199 findings (≈75 dead legacy anchors from tasks pointing at retired question ids; ≈54 completed clean-results in no question's evidence; relates_to ⇄ evidence mismatches incl. #722/#763/#494–#501; spec-role-header State trailer stale vs 7 promoted results).
  • Drift hash 078285c14444 — byte-identical to the 07-07 and 07-22 rows.
  • drift proposal already open (logged 2026-06-29/07-04/07-06/07-07); not re-proposing

My thoughts

Highlighted results

  • no results promoted today