EPS
← All docs·Activity

Daily — 2026-07-01 (fleet-scale workflow-fix day; 0 promotions)

updated: 2026-07-04

What happened

  • Heaviest pipeline day yet: 1,988 workflow markers across 61 tasks (#658–#824), ~2,350 commits on main, 0 clean-results promoted (5 tasks newly parked at awaiting_promotion: #778, #779, and the #810 family in flight).
  • Science line: #778 (Persona Vectors replication + null battery) ran end-to-end on a RunPod 8×H100 after a GCP H100 stockout in both us-central1 zones — parked awaiting_promotion ("persona vector fails the null battery" landed in INDEX.md); #779 (trait-training vs learned map) parked awaiting_promotion; the #810/#811/#812 answer-summary sweep was designed in the PM session, filed, and all three autonomous sessions launched; #813 (map-change substrate dependence) was spec'd out of a chat-surfaced methodology gap (base-map vs map-change run on different, never-reconciled substrates) and spawned; #823 filed; #667 got a Thomas-driven all-layer Δc + map-change deep dive plus a tf-margin follow-up round (wound down PARTIAL after 5 pivots).
  • Workflow-fix wave: ~30 kind: infra workflow-fix tasks (#774–#824 range) filed and mostly merged same-day — headliners: #815 (repo-root git reset --hard ban + lint, after a #778 analyzer's reset clobbered #812/#813 task state), #784 (review-round cap 3→5, Thomas-directed), #802 (code-reviewer bug-class sibling sweep — Thomas's "fix the whack a mole"), #817 (persist-by-default upload policy, from the #779 lost-rollouts postmortem), #783 (GCP queue-timeout → RunPod failover), #821 (pods.conf never-drop-RUNNING).
  • All 28 .claude/agents/*.md repointed to model: claude-fable-5 at xhigh effort (99c7463bc6).
  • Notable failures: earlyoom killed #742's 9-hour judge rerun with no resumable state (all 5,520 API results lost; persistence rebuilt with per-cell checkpoints same-day) and #761's bootstrap fit twice; a GCE instance cloned main instead of issue-812 (→ #824); 8 GCP/RunPod/Anthropic transient failures recovered autonomously.

Applied workflow improvements

Backfill context (run 2026-07-04): 07-01's own sessions filed ~30 workflow-fix tasks same-day, and the 07-02/07-03 backfills filed #997–#1023 — this sweep routes only what remained genuinely uncovered against the CURRENT tree. Skipped-as-already-fixed: the issue744_dump_and_stream.py bare-dotenv lint failure that taxed ≥11 sessions (lint now PASSes on main; the process-side fix #1022 is in planning), multi-pending epm:followup-scope selection (SKILL.md now scans ALL entries), the nested-worktree --show-toplevel trap (gotchas bullet already present), pod-side clean_experiment_downloads hang (#803), and the spawn/dispatch-race cluster (#843/#844/#956, landed 07-02).

  1. Target: CLAUDE.md (§ Routing experiment intent → NEW direction) — what: filing a task from chat is not an execution greenlight; never auto-spawn a just-captured task — commit: 8f8b8e4d2c Why: #823 capture (62c6e71a session, 23:38–23:43Z): the assistant filed the task AND spawned an autonomous session; Thomas immediately: "Wait can you stop it". Task capture ≠ run approval. Applied edit:

    - ... Create `status='proposed'` task; only execution path is `/issue <N>`.
+ ... Create `status='proposed'` task; only execution path is `/issue <N>`. **Filing ≠ spawning:**
+ capturing a NEW-direction task from chat does NOT imply launching it — do not auto-spawn a
+ `spawn-issue --auto` session for the just-filed task unless the user explicitly asks to run it ...
+ (Incident #823, 2026-07-01: an auto-spawn immediately after capture drew "Wait can you stop it")

    Verification: workflow_lint.py --check-references + --check-asks PASS repo-wide after all commits.

  2. Target: .claude/rules/gotchas.md — what: pgrep/grep/pkill self-match bullet (false liveness, false marker-wakes, self-kills) — commit: 630be6c3bf Why: three hits on 07-01: a pgrep -f liveness probe matched its own cmdline and hid a dead launch ~50 min ("dispatch/workers alive: 2"); a poll's grep -q 'epm:failure' woke on the string inside a reconciler note; pkill -f 'watch_778_followup.sh' killed its own compound command (exit 144). Applied edit:

    + - **`pgrep -f` / `pkill -f` / a log-`grep` liveness probe can match the PROBE'S OWN command line ...**
+   RULE: anchor process patterns to the executable (`pgrep -x`, `pgrep -af '^...'`), exclude self,
+   and read markers structurally (parse the `kind` field), never substring-grep a JSONL for marker kinds.

    Verification: lint gates PASS (above).

  3. Target: .claude/skills/daily/SKILL.md (route-2 filing step) — what: explicit Bash timeout / small batches when filing many tasks — commit: f3746d665b Why: the 06-30 backfill run (92f20727, 06:54Z on 07-01) had its 16-task sequential filing loop killed at the default 2-min Bash cap mid-loop, forcing a which-got-filed audit. Applied edit:

    + When filing MANY tasks in one run, give each `file_infra_task.py` Bash call an explicit `timeout`
+ (>=300000 ms) or file in small batches — a 16-task sequential filing loop under the shared registry
+ flock exceeded the default 2-min Bash cap mid-loop (2026-07-01, the 06-30 backfill run).

    Verification: lint gates PASS (above).

  4. filed for review #1024: parse_judge_json drop-never-coerce (src/explore_persona_space/eval/utils.py) — the shared binary-judge parser still coerces malformed judge returns to a default (llm-judging rule 9 violation; #766 fixed only belief.py). Evidence: ~9% of 87,055 judge calls on #778 coerced (~30% in the first tranche), plus the same WARNING on #763's run log the same day.

  5. filed for review #1025: codex-critic numeric-leak guard strips cross-issue #N refs — during #795's critique the guard stripped the #720 reference, so Codex-Statistics could not weigh the duplication evidence that later made #795 a no-change deflection. Whitelist task-reference tokens.

  6. filed for review #1026: an in-flight upload-verifier must BLOCK finalize/status-advance — #778 flipped to interpreting and finalized the pod on an "agent-level PASS fallback since no artifact declaration" while the verifier was still running; its v2 verdict came back FAIL minutes later (a benign marker-string defect this time — with a real upload gap the pod would already have been gone).

Other problems & notes

  • Route 3 (needs you): nothing NEW filed — the day's judgment items are already tracked: registry/filesystem drift (#801), ruff-debt burn-down (#1023), the pod-667 stranded-data wait-vs-regenerate call (07-02 daily), and the open living-docs re-synthesis proposal (below).
  • The day's defining incident — concurrent destructive git: a #778 analyzer subagent ran git reset --hard origin/main on the shared repo root, clobbering ~10 of #778's markers and #812/#813's task state (recovered 81c52d6a2b/d29a877e6f; #812's body restored next evening). Same-day response: #815 (CLAUDE.md ban + --check-no-repo-root-git-reset-hard lint) landed; the related shared-main hazards (#786 atomicity, #787 Step 10d merge machinery, #796 guard detach coverage, #804 clause-local guard) all merged the same day. Residual sibling frictions recorded: a concurrent autostash pop leaked foreign edits into a worktree (#792 session); one session's stale-folder cleanup ran before pushing its status commits, briefly corrupting the origin REGISTRY pointer for #808 (self-recovered).
  • Thomas-correction behavioral notes (delivery, not absence): silent dispatcher death on #667 went unnoticed ~2.5h until three "check progress" pings + "why didn't you rerun on your own?" (the armed bg poll never fired; hardened in-session); over-gating #811/#812 behind #810 on a fragile cron ("Why can't you just spawn them now" — "Fair — I over-gated"); a "definitive" wrong-frame answer on the c_C recipe held until "are you SURE?" — verification then surfaced the real substrate gap that became #813; a near-duplicate analysis proposal ("Did we not already run these analyses? Search deeply" — most items pre-existed in #667 A3.7); two quantitative retractions under pushback (double-subtracted install metric; "negatives dominate" vs the exact 200:200 mix); a stale-status answer ("Are you sure 802 isn't already running" — it was completed); "No I mean what is 779"; and the #823 MLP-training rejection ("Don't do any MLP training") + premature spawn (route-1 fix 1 above).
  • Vectorize-first class fired 4× in one day (#722/#778/#823 lineage): #667's inline map-change fit stalled serial (~20 min on cell 1/84 — batched same-day via vectorized_mlp_skill.py); #763's autonomous session resolved an 80× compute deviation as continue_as_is while holding an idle 1×H100 ~16h — the PM killed both duplicate sessions, stopped the pod, and re-drove it to vectorize; #810's implementer did it right (posted epm:compute-deviation, 308× → 4.2× with n_perms kept); a #779 subagent ran a serial layer sweep with a self-built OOM-looping restart watchdog (4+ OOMs, 50 blind retries) that Thomas caught ("vectorize and rerun"). The rule catches at review/deviation time, not authoring time; the 07-02 vectorization cluster (#869–#876) since hardened this.
  • Science flags carried on task records: #778's capture judge coerced ~9% of calls (→ #1024); #778's monitoring leg used the 10 extraction instructions instead of the paper's 8 eval prompts (partly tautological overall_r; corrected-monitoring follow-up + #816 spawned same evening — #816 later on_hold per the 07-02 audit); two factual errors in the live #779 clean-result survived the critic gates until Thomas probed (40-vs-20 eval questions; a "disjoint corpus" Design claim that was actually LOCO); #667's all-layer extraction aliased the L0 slot to L14 data (L0 dropped from all reads); #813's rsLoRA parity floor false-failed a known-good weak-writer adapter (PARITY_MIN_WRITE_RATIO=0.01 vs 0.00903 — recalibration crash-fix round in flight at midnight, agent memory written).
  • Data-preservation tensions: #667's tf-margin round wound down PARTIAL with 32-cell extract data on stopped pod-667's volume, and pod-667 resume then failed 7 consecutive SUPPLY_CONSTRAINT cycles with ~53K all-layer files unpreserved (no GPU billing; still the open wait-vs-regenerate call noted in the 07-02 daily). #779's lost extraction rollouts drove #817 (persist-by-default) which merged same evening.
  • Cross-family review earned its keep: Codex twins caught substantive bugs Claude reviewers PASSed in ≥13 rounds across the day (e.g. #763's snapshot_download sibling misses, #787's push-before-merge gap and clobber-prone sub-fix — all 3 Claude critics APPROVEd, all 3 Codex REVISEd, 3 reconcilers upheld Codex —, #812's batch custom_id :: that would have killed the ~600K-call production batch, #667's dispatcher-default override, #786's stale-branch file deletions, #778's silently-dropped bare-int judge scores). Flip side: the whack-a-mole loops this generated drove #784 (cap 5) + #802 (sibling sweep), both landed same-day.
  • Codex runtime flakes (recurrence): 2 no-shows → single-reviewer fallback (#815 r1, #792), one stream-idle-timeout retry (#814), sandbox HF/DNS blocks producing procedural-only verdicts (#779 r3 ×2) — the #789 sandbox-unverifiable advisory tag landed same-day; result-fetch retry is #1020 (in planning).
  • Fleet/VM pressure: earlyoom --prefer '(^|/)(pytest|python3?)$' makes every uv run python workload the preferred kill victim — killed #761's fit twice and #742's 9h judge rerun (root-caused in-session; .venv/bin/python3.11 comm evasion + per-cell checkpoints; the durable thread-caps/earlyoom fixes are the 07-02 #847/#848/#849 cluster); VM load ~18-45 SIGTERM'd probes and wedged a helper agent; 62 Happy sessions found, 31 zombie wrappers reaped (~5.2 GB RSS) → #818 filed + implemented same-day; autocompact thrash killed ≥5 heavy subagents in #820's session alone (07-03 root cause: All-tools spawns load ~138K MCP-schema tokens → analyzer-lean twin + the 07-02 trim cluster).
  • Session-collision residue (guards mostly held): ~6 clean Step-0 single-orchestrator guard exits; but #806 had TWO sessions run the full pipeline to completion in parallel, #805's registered session was raced by an unregistered headless one (its 3 Codex REVISE concerns were never formally folded in), #787 was watcher-double-dispatched 37s apart, and #778's watcher respawn created a duplicate driver that had to yield mid-round. The dispatch-lease/registration fixes are the 07-02 #843/#844 (+#956) cluster.
  • Note-only recurrences (hooks/guards worked; friction only): sleep-chain waits blocked ≥8×; bare | python -c pipes blocked ≥11×; Write/Edit-before-Read rejections ~11× (one cause: interleaving sed -i with Edit on the same file); --note+--file mutual-exclusion argparse trip; hand-built stale tasks/running/800/ path; gh pr view --json merged unknown-field; deferred Monitor called without ToolSearch (06-30 backfill session); post-merge stale-task-folder guard fired correctly 4× (#787/#797/#784/#790).
  • Watcher respawn-cap dead-end (observed, mechanism worked): #722 sat orphaned at followups_running with the daily respawn cap exhausted (2/2) — the watcher's alert reached the PM, which absorbed the work into #810–#812. Two #664 user-pre-authorized follow-ups had been silently dropped at session park; Thomas surfaced it, re-driven same day (the systemic binding fix is #820's family).
  • Backfill provenance: the 07-01 nightly /daily died unwritten (the #994 headless bg-wait class, since fixed); this file was backfilled 2026-07-04 from 122 transcripts with in-file 07-01 timestamps, mined by 6 parallel subagents; this run is itself #994's backfill-acceptance evidence (detached headless, pid 112669).

Living-docs drift

living_docs.py check FAILs with 203 findings — byte-for-byte the drift state already proposed for re-synthesis by the 07-02 backfill (event-stream row d2a80db46b3b, logged 2026-07-04, superseding the still-open 06-29 proposal 1736fca0804f):

  • 4 registry/filesystem inconsistencies: #696, #750, #764, #766 (tracked as held task #801; repair is task.py audit --repair --apply).
  • ~75 dead legacy relates_to anchors (a1–h3 scheme, tasks #28–#411); ~60 relates_to ⇄ evidence mismatches (incl. #722, #763 vs leak-predictor/identity-cb-duality); 54 completed clean-results in no question's evidence; spec-role-header State trailer stale vs 7 later results.
  • drift proposal already open (logged 2026-07-04); not re-proposing.

Parked-proposal re-consideration: #713's living-docs rejection is an infra task with no context change (skip); #809 (parked-redundant, duplicates #778) stays parked — #778 is still unsettled pending its honest-nulls rerun. No revivals to surface.

My thoughts

Highlighted results

  • no results promoted today