EPS
← All docs·Activity

Daily — 2026-07-22 (compute-starved day;

updated: 2026-09-03

What happened

  • #779 landed its follow-up round and a user-authored sibling. The l26-kernel-gate-recovery same-issue round completed → awaiting_promotion (PR #1389 merged, methodology doc exported); the persona-vector pre-image sanity check was saved as new clean-result #1615 (straight to awaiting_promotion); the prefix-based mapping twin + a 1000-draw random-direction null ran as inline rounds (the null materially qualified the selectivity claim: evil p=0.015, hallucination p=0.021, sycophancy marginal).
  • #1586 (matched-install LoRA vs full-FT) was the day's saga. Built + review-PASS by 09:30Z, then blocked ~10 h by a GCP DWS drought stacked on a broken RunPod lane (VM SSH key dropped by the fellows onboarding + the team burn-cap counting the fellows fleet's ~2.8k/hr).Thomasoverrodeat19:09Z("justrunonrunpod");dualpodstrainedthefull16−cellgridthrough7crashes/6reviewedfixrounds(disjointness−assertfalsepositive,ENOSPCfromtrain−all−before−persist,marker−seamKeyError,HFLFS403,vLLMengineinit);bothpodscompleted,upload−verified,terminatedby 04:49Z. 2.8k/hr). Thomas overrode at 19:09Z ("just run on runpod"); dual pods trained the full 16-cell grid through 7 crashes / 6 reviewed fix rounds (disjointness-assert false positive, ENOSPC from train-all-before-persist, marker-seam KeyError, HF LFS 403, vLLM engine init); both pods completed, upload-verified, terminated by ~04:49Z. ~395 vs the $200–300 estimate.
  • Massive same-day workflow-fix wave: ~15 infra fixes merged, several fixing problems observed live that morning — #1596/#1600/#1601 (burn-cap scoping + queue-vanish/timeout on-demand retries), #1610–#1612 (from the #1586 crash deep-dive), #1603 (exit-75 re-drive), #1605–#1607, #1590–#1595, #1598–#1599, #1604 (mapping-baselines mandate wired into planner/critics), #1613. #1609 merged the fellows Slurm lane (fellows-first DEFAULT_AUTO_LANE_ORDER) and is running a fast-start-guard follow-up.
  • #1092 consolidated prefix/query/context→answer writeup iterated heavily with Thomas (novelty + clarity corrections), plus several inline rounds — the spread-delta round's raw-L2 null flipped to ρ≈+0.9 under the established #658 whitened recipe; #1112's LoRA-vs-FT arms finally ran after a ~14 h drought (the 1× shape had stock all along); #1415 round 2 merged (PR #1388).
  • Codex org quota exhausted (until ~2026-08-06): every doubled review site ran single-Claude all day; the #1204 pre-spawn skip worked cleanly everywhere (no wasted spawns).

Applied workflow improvements

Route-1 self-applied (7 — each independently revertable; repo-wide lint gate --check-references + --check-asks PASS after all commits):

  1. Target: CLAUDE.md — what: mapping-baselines bullet: replaced the stale "explicit wiring … rides the filed workflow-fix task" tail with the landed #1604 enforcement surfaces (planner.md §6, critic-lens-reference item 15, statistics-critic item 15, experiment-guidelines g11) — commit: c9a8747b28 Why: parked #1604 candidate (routed via the Step-C sweep); #1604 merged 4c4e79688b earlier today, leaving the always-on rule pointing at a task that already landed.
  2. Target: .claude/rules/gotchas.md — what: new entry: np.savez appends .npz to any non-.npz target, so *.npz.tmp atomic-rename checkpoints crash FileNotFoundError after the compute finishes — commit: 611e0b5a39 Why: #1092 shrinkage-refit (session 5e8b4c66, 23:22Z) lost a ~20-min refit to exactly this.
  3. Target: CLAUDE.md — what: index.lock recovery wording: retry-once extended with a bounded poll (~60 s) under heavy fleet activity — commit: 42f617cc3f Why: three sessions needed ad-hoc poll loops in one evening (5e8b4c66 needed 15 iterations; retry-once was insufficient).
  4. Target: .claude/skills/paper-plots/SKILL.md — what: decision-table row: pin ONE color↔meaning assignment across every figure in a writeup; never reuse the palette pair for a different factor — commit: 79fb5d104d Why: two #1092 figures used the same two colors for model identity vs arm; user: "it says green is base model and blue is instruct model ???"
  5. Target: .claude/rules/upload-policy.md — what: two quota-403 corollaries: (a) an unblock probe must exercise an LFS-SCALE upload (small files pass while LFS is still 403 — a small-file probe minted a premature billing-resolved marker on #1586, costing an extra crash round); (b) user-action escalations name the exact click path + expected end-state — commit: d687438291 Why: #1586's billing saga: one avoidable crash round + ~1.5 h of user back-and-forth on a billing page that was already correct.
  6. Target: .claude/skills/issue/SKILL.md — what: 9a-ter clarification: the inline lint-gate cert binds at COMMIT time (the hook validates there, not at push); preferred ordering = kick the gate off in background at script-freeze — commit: 722e137c54 Why: ~5 guard-block events across 3 sessions from reaching the commit before the gate.
  7. Target: .claude/rules/selection-symmetric-nulls.md — what: primary vs record-only labeling: a known-broken companion convention is never presented co-equal with the valid one — commit: 0ba4e76238 Why: #1415 writeup co-headlined the broken shared-baseline statistic; two user challenges before the split was stated.

Route-2 filed for independent review (19 — each dispatches through the full planner/critic/code-review pipeline; ids from logs/daily/filings-2026-07-22/filed.jsonl):

  1. filed for review #1616: register test_issue_skill_width_reeval_pointer.py in WORKFLOW_INVARIANT (SKILL.md-diff leg unguarded; semantic probe confirmed the #1496 rules-pin arm covers only the rule-file legs) — from the parked #1594 candidate.
  2. filed for review #1617: gcp.py loud marker on boot-disk reuse + size mismatch (the #779 20-day-old 300 GB disk silently reused) — from the parked #1602 candidate.
  3. filed for review #1618: research-pm.md 47,861 B > 47,000 B ratchet cap — tests/test_workflow_lint.py red on pristine main fleet-wide — from the parked #1611 candidate.
  4. filed for review #1619 (wf_fix: false, experiment code): three issue1092 fig scripts import numpy/scipy before load_dotenv(), failing test_no_new_torch_before_dotenv_vm_entrypoints on main — from the parked #1613 note.
  5. filed for review #1620: guard_root_code_commit.sh blocks pathspec-limited commits on a CONCURRENT session's staged uncertified file (≥8 block events / ≥4 sessions today; drove 3× EPM_ALLOW_ROOT_CODE_COMMIT overrides) + one flaky fresh-cert block — the day's top recurrent friction.
  6. filed for review #1621: guard_repo_root_branch.sh false positives — the sanctioned git worktree add --no-checkout --detach recipe, and heredoc PAYLOAD text quoting git checkout -b.
  7. filed for review #1622: project CLAUDE.md ad-hoc bullet gains the artifact-turn-ends-with-verified-bare-URL clause (~5 violations today; the global rule alone is not holding).
  8. filed for review #1623: ad-hoc similarity statistics declare direction-aware vs spectrum-only + name the precedent battery (the #1310 Procrustes spectrum-cosine overclaim).
  9. filed for review #1624: workflow_lint check flagging bare list_repo_files on the ~1M-file data repo (two >7-min wedges in one day despite the documented scoped-listing rule).
  10. filed for review #1625: inline rounds state BOTH mapping arms (prefix + context) in the dispatch note + Read-check rendered figures for non-empty axes (context-only mapping user-caught on #779; empty figure pasted 3× on #1112).
  11. filed for review #1626: workflow_lint root-guard-hook probe treats transient rc=-1 as retry-once-then-WARN, not a blocking FAIL line (#1610 gate flipped to block on it).
  12. filed for review #1627: Step 9c/10d gate single-flight guard + process-exit-keyed Monitor (#1606: duplicate gate runs racing, 2 fail-CLOSED blocks, ~12 min churn).
  13. filed for review #1628: GCP teardown handles DWS-PENDING instances (delete timed out ~4× over 10 min during the #1586 pivot; --async errored).
  14. filed for review #1629: /issue-tick no-ops in sessions with recent human activity (a tick fired into Thomas's live Q&A; he had to interrupt it).
  15. filed for review #1630: /daily commits stage only their own explicit paths + verify the staged set (last night's daily commit 7dbde267f1 swept the #1092 session's 4 uncommitted files onto main).
  16. filed for review #1631: committed plan_patch.py helper (normalized-anchor matching) replaces improvised plan-edit scripts (≥4 first-try anchor failures across 3 sessions today).
  17. filed for review #1632: exclude /tmp/ scratch from the PostToolUse ruff-format hook (it reshaped a just-Written edit script and broke the Edit anchor).
  18. filed for review #1633: down-width split guidance — plans name each arm's minimum runnable width; probe the narrower shape on a sustained stall (#1112's 1×-runnable arm sat ~14 h behind a 4×/8× queue while 1× had stock).
  19. filed for review #1634: pod-side pid files are populated from the launch expression ($!), never a post-hoc pgrep (transient-pid capture caused 2 false "exited" alarms on #1112).

Other problems & notes

  • Route 3 (needs-human, tracked proposed tasks the PM will surface):
    • #1635 — inline-round scientific discipline (scientific-meaning carve-out): three same-day incidents — the #1092 raw-L2 spread null that flipped to ρ≈+0.9 under the established #658 whitened recipe (wrong-signed conclusion lived ~2.5 h); the #779 correlation headline folded into a writeup before any null existed; the #1092 writeup restating banked #722 results as novel PLUS its "not the behaviorally relevant part" claim still unqualified against its own base-model counterexample (checked at filing: docs/results_summaries/2026-07-22-direct-vs-averaged-prefix-map.md lines 12/54). Decide which duties to mandate + greenlight the base-arm qualification edit.
    • #1636 — shared-root inline-round hygiene (ambiguous-intent carve-out): a concurrent worktree revert wiped an in-progress #1092 edit round mid-commit (near-miss loss); pick claim-markers vs scratch-worktree default for multi-file inline rounds. Note: scripts/issue1092_fair_deepdive_figs.py + figures/issue_1092/prefixend_monitoring_averaged_only.* were still modified-uncommitted at the shared root at mining time — this run did NOT touch them (owning-session liveness unknown).
    • #1637 — corpus-capture redaction default (data-handling judgment): flagged LMSYS rows stored as category placeholders surfaced as placeholders in a user-facing dashboard; Thomas had to ask twice. Decide truncated-text+flag default vs placeholders.
    • already tracked in #1472 (shared: codex/quota/single-claude tokens): the Codex-quota-outage hold (out until 2026-08-06; every doubled review site single-Claude all day) — duplicate filing suppressed (#1483); freshness epm:progress note posted on #1472.
  • Skipped filings (premise gone at verify time — 5): (a) runpod_api.py:488 UP037 ruff red — ruff passes clean on the current tree; (b) RunPod SSH-key preflight — superseded by b66910d748 (PUBLIC_KEY injected at pod create; provision no longer depends on account-key state); (c) RunPod burn-cap scoping — landed same day as #1600 (EPM_RUNPOD_BURN_SCOPE=managed); (d) smoke-must-cover-all-phases policy — landed same day as #1611; (e) fellows-cluster memory-note correction — the stale-endpoint root cause was found and the memory updated later the same night (01:35–01:54Z, session b2b4d655).
  • Retraction re-check: 5 marker-mined candidates scanned (Step C: #1594/#1602/#1604/#1611/#1613), 0 retractions found; 1 sub-item skipped on a current-tree check (the UP037 red), and 4 transcript-mined candidates were dropped at verify time as already-landed (list above). Transcript-mined items were tail-checked by the miners through end-of-transcript.
  • RunPod burn-cap typed-reason note: exit-75 burn-cap refusals were surfaced to the user as "supply-constraint" in a #1586 brief; with #1600's managed-pod scoping the fellows-fleet trigger is gone, so a distinct runpod_burn_cap_refused reason code was NOT filed (largely superseded; revisit if a genuine EPS-spend refusal gets mislabeled).
  • Spurious usage-policy refusal kills: ~6 recovered, 0 lost work — /issue 1598 died turn-1 (respawn machinery recovered); #1605 code-reviewer (rung-b rephrase PASSed), #1602 fact-checker (thinned respawn), #1592 fact-checker, one #779 output-file read, one #1606 implementer mid-stream death (thinned respawn). All per the documented ladder.
  • Adherence-lapse recurrences (rules exist; no new gap filed): log-rotation-on-relaunch skipped on a VM-local inline phase (2 false crash alarms, #1092); choom/checkpoint protection applied only on retry after an earlyoom kill (#1092 spread-delta, ~18 min lost — 9a-ter already mandates it); 3 stale assignment re-deliveries to teammate xpersona-1310 (durable-state probe skipped); verify-before-asserting lapses ×3 (the #810 "reconstruction gains" overclaim, the n1M all-28-layers scope-note error, #1611's incident-text ordering error — the last caught by code review); piped-git/sleep/premature-commit guard blocks ~7 events, all correctly caught.
  • Sent≠seen recurrence: the #1586 capacity chat (0b9be230) went dark on Thomas's 22:23Z "just override and run on runpod" — no reply ever, and the override had already been executed 3 h earlier via another surface; he re-issued a stale directive.
  • User figure/writeup corrections cluster (#1092 writeup): pooled-vs-split populations took 3 figure rounds (~40 min); ~15 clarity queries; "remove any reference to direct prefix"; "plots NOT tables"; behavioral note — default to the fully pooled single-series figure and lead with the question, not job plumbing.
  • Title-sync drift sweep (Step D): title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-11.md for the verbatim rows).
  • Daily-stub push race (this run): first push of the route-1 commits rejected (concurrent committer), recovered by sync_repo_root.py (state=synced, push landed).
  • Minor notes: one workflow_lint rc=124 timeout inside an inline gate (re-run passed; single event); a goal-drift md5 false positive from a hand-rebuilt snapshot (#1614, ~25 s); a c40 stale plan-header WARN (#1604); one non-fail-fast gate chain where a failed link didn't block the persist (#1591, self-caught); GCP delete + HF billing sagas otherwise resolved in-session; the p9 stager's manifest.json glob miss was caught twice by the upload-verifier (experiment code; safety net worked).

Living-docs drift

  • living_docs.py check: FAIL (RC=1) — 199 finding lines, drift hash 078285c14444 IDENTICAL to the 07-07 row: ~75 dead legacy anchors, ~54 completed clean-results in no question's evidence, the relates_to⇄evidence mismatches (#722/#763/#494–#501), and the stale spec-role-header State trailer.
  • drift proposal already open (logged 2026-06-29, extended 07-04/07-06/07-07/07-19/07-20/07-21); not re-proposing — event row appended to .claude/cache/nightly-consolidation-events.jsonl.
  • Parked-proposal re-consideration (Step B): no un-reconciled epm:living-docs-update-rejected markers and no on_hold parked-redundant follow-ups — nothing to re-surface.

My thoughts

Highlighted results

  • no results promoted today