EPS
← All docs·Activity

Daily — 2026-07-08 (Step C backlog cleared — 71 filed; 21 infra fixes merged; 3 science rounds; 0 promoted)

updated: 2026-09-03

What happened

  • #1092 (realistic sparse-crossed corpus) ran its full pipeline end-to-end. Launches 5→9 on pod-1092 (8×H100) through crash-fix rounds 8.5–8.9 (vLLM H100 illegal-memory-access mitigation, bf16-null gate recalibration, dependency-DAG fixes with an all-8-cell tiny smoke, rendered-length pair-drop filter); all 8 cells generated + captured, pod terminated 16:51Z. P5 judge finished off-pod (36,357 items × 3 traits, 27 dropped rows). P6 fit grid: VM pilot earlyoom'd (real RSS ≥22.9 GB vs 8–10 projected) → routed to the pre-registered cpu-bigmem GCE lane; pilot healthy at day end. Live watch item: the P6 pilot's wall projection may bust the 30 h --max-run-duration fence (launched 23:40Z) — surface before the fence eats the grid.
  • Three science rounds folded and re-parked at awaiting_promotion: #1112 lr-matched-method-pair (at matched lr 5e-6 + matched install, full-FT still shifts the residual stream far more than LoRA: paired ‖μ‖ diff +4.23 [3.92, 4.54] @ L14, MODERATE); #1090 fu3 posonly-contexts-parallel-matrix (new headline: the training context, not the contrastive-negative panel, gates where implanted behaviors express and leak — registered falsification branch FIRED; MODERATE, one seed); #825 round-8 sampled-separator control (round-7's gap-closure is greedy-specific: D 0.590→0.031 base / 0.428→0.086 instruct under sampling; MODERATE). #928's cheap-band round 2 planned + approved. The awaiting_promotion queue now holds 91 tasks; nothing was promoted today.
  • 21 infra/workflow-fix tasks completed + merged in one day (#1125–#1147 band, all from yesterday's /daily filings + same-day follow-ups): repo-root merge fence (#1128), refusal-wedge turn-level predicate (#1127), Codex quota sentinel (#1126), /daily retraction re-check (#1131), the Step C parked-candidate sweep this very run used (#1132), per-file pristine-oracle timeouts (#1129), rc==0 stderr forwarding (#1130), bare push/merge snippets (#1138), issue-1090 stranded fleet fixes incl. the CVD-pin (#1144), torch-before-dotenv offenders (#1145), thread-caps predicate (#1146), Step-10d invariant tests (#1147), + gotchas/rules entries (#1133, #1135, #1136, #1137, #1139, #1142, #1143).
  • Codex org quota exhausted all day (until 2026-08-06) — every doubled review site fleet-wide ran the documented Claude-only fallback; the #1126 sentinel short-circuited dispatches but composer subagents were still wastefully spawned in most sessions (filed as #1204). Decision task #1140 (pay vs ride it out) still needs Thomas.
  • Refusal pressure continued on #1092/#1090: one orchestrator wedge (4 refused turns, session died 14:20Z, 33 min lost), one respawn refusal-killed on its FIRST turn (~100 min recovery gap — filed #1209), ~6 subagent kills incl. two refusal-truncated Agent spawns that dispatched cut briefs (recovered via SendMessage; codified tonight as CLAUDE.md clause (g)).
  • First run of the /daily Step C parked-candidate sweep (#1132) cleared the whole 109-candidate backlog (June 30 → July 8): 43 already fixed on main, 2 deduped, 2 out-of-scope, 4 merged into siblings, 58 still-live candidates filed for review, 4 held as needs-human. All 109 got routed-record markers, so future sweeps skip them.

Applied workflow improvements

  1. Target: CLAUDE.md — what: refusal-recovery bullet gains clause (g): a mid-stream refusal can dispatch a TRUNCATED Agent spawn; verify prompt completeness + complete the brief via SendMessage — commit: e22b66b0e1 Why: #1090 session (c16b10ca, 08:43–10:46Z): two Agent spawns dispatched cut mid-sentence in the same second as their refusals; the orchestrator noticed and completed both briefs via SendMessage parts — a twice-proven recovery no rule named. Applied edit: 
    - ...governed by (b)/(e) above — a LOWERED refusal surface, not immunity.
+ ...governed by (b)/(e) above — a LOWERED refusal surface, not immunity; (g) **refusal-truncated Agent spawns:** a mid-stream refusal can still DISPATCH a partially-generated `Agent` tool call — the subagent starts running on a brief cut mid-sentence (#1090, 2026-07-08: ...) — so after any refusal on a turn containing an Agent spawn, VERIFY the dispatched prompt's completeness and, if cut, complete the brief via SendMessage...
    Verification: doc-only; workflow_lint.py --check-references + --check-asks PASS repo-wide after commit.
  2. Target: .claude/rules/plan-compute-sizing.md — what: #1092 incident line in the CPU-phase RAM/RSS routing gate: fit-grid RSS projections presumed underestimated until pilot-measured on ONE unit; a projection within ~3× of the VM bar routes straight to cpu-bigmem — commit: a8cfaf7048 Why: #1092 (8fdf2c4c, 18:45Z): P6 pilot's real per-unit RSS ran ≥22.9 GB vs the plan's 8–10 GB (~2.3–2.9× under); earlyoom SIGTERM-killed it on the shared VM before the pre-registered escape lane recovered it. Applied edit: 
    + ; #1092 2026-07-08: a fit-grid pilot's real per-unit RSS ran ≥22.9 GB against a plan projection of 8–10 GB ... a fit-grid RSS projection is presumed underestimated until pilot-measured on ONE unit at production shape, and a projection within ~3× of the VM routing bar routes straight to cpu-bigmem
    Verification: doc-only; lint gates PASS.
  3. Target: scripts/guard_repo_root_branch.sh — what: BLOCKED message now says the guard matches command TEXT not cwd, so a worktree-internal op after cd <worktree> is still blocked — use git -C <worktree> — commit: 72a1263789 Why: #1143 (fa0fb96a, 12:06Z): a compound did cd "$WT" before a checkout and was blocked; the agent had to infer the git -C escape (~1 wasted turn). Applied edit: 
    + This guard matches COMMAND TEXT, not cwd — a worktree-internal op after 'cd <worktree>' in a compound is still blocked; use the git -C <worktree> form instead of cd'ing (incident #1143, 2026-07-08).
    Verification: bash -n PASS (message-text-only change); lint gates PASS.

Filed for review — Step C parked-candidate sweep (56 tasks, wf-fix + daily-auto-filed, → /issue --auto / watcher backstop). First run of the #1132 pass over the 109-candidate recursion-guard backlog (2026-06-30 → 07-08); every candidate was current-tree-verified and retraction-checked by 6 triage subagents before filing:

  1. filed for review #1148: /daily SKILL's surviving hand-rolled repo-root pull recipe → sync_repo_root.py (from #1047)
  2. filed for review #1149: mechanical ensemble reviewer no-show predicate (from #856)
  3. filed for review #1150: forward rc==0 stderr at the watcher set-status + filer sites (from #1130)
  4. filed for review #1151: GCP crash-persist no-fire on #811-class crashes — investigate + harden (from #1055)
  5. filed for review #1152: harden guard_harmful_bank_read hook (3 reviewer-surfaced bypass shapes) (from #965)
  6. filed for review #1153: lint pinning the awk elision program identical across its two homes (from #998)
  7. filed for review #1154: lint marker-recipe doc numeric snippets against code constants (from #1001)
  8. filed for review #1155: pod-safety stop-failed marker + fix the never-kills guard-test span (from #980)
  9. filed for review #1156: poller WARN when the pid file is older than the run-launched marker (from #1070)
  10. filed for review #1157: 6d.0-bis smoke gate names data-ingestion/real-corpus tiny-real probes (from #1113)
  11. filed for review #1158: recovery rule for autocompact-thrash subagent deaths (from #1115)
  12. filed for review #1159: split the clean-result-critic.md monolith + pointer-coverage lint (from #850)
  13. filed for review #1160: fix trunk-red dotenv-before-torch test — 2 issue779 offenders + stale anchor (from #1079; also flagged live-red by triage)
  14. filed for review #1161: raise-by-default _hf_tree_get guard in verify-body tests (from #860)
  15. filed for review #1162: check_jsonl_splitlines generic-receiver + assignment-dataflow signals (merged #1132+#1032 candidates; the U+2028 shred reached code review today)
  16. filed for review #1163: workflow_lint no-flags run times out from a worktree cwd (from #1019)
  17. filed for review #1164: mechanize the reused-artifact realized-keys probe (from #1087)
  18. filed for review #1165: venue caveat in the experimenter nohup-.env memory (from #984)
  19. filed for review #1166: planner.md verdict-lattice declaration form (from #1006)
  20. filed for review #1167: per-tick push cap in the watcher triage observer (from #967)
  21. filed for review #1168: forward-port pin-first figure reads to the v2 reviewer specs (from #1056)
  22. filed for review #1169: document the stale-blocked watcher pass + GC test fixture (from #1021)
  23. filed for review #1170: observer for unreconciled doubled-site verdict disagreements (from #1134)
  24. filed for review #1171: body audit strips v4 ## Methodology sample blocks (merged #969+#970 candidates)
  25. filed for review #1172: launcher-level PYTHONPATH export for the managed GCE/SLURM lanes (from #853)
  26. filed for review #1173: /daily route-2 bodies carry the workflow_fix_target: Provenance line (from #1134; tonight's filings already carry it)
  27. filed for review #1174: relative cache exclusion in the improver-spawn lint (from #1012)
  28. filed for review #1175: lint unguarded deferred scripts.* imports (from #853)
  29. filed for review #1176: lint SKILL.md fenced git recipes against the root guard (from #1047)
  30. filed for review #1177: provision-time WARN on terminal-parent pods (from #1080)
  31. filed for review #1178: post-marker WARN on literal backslash-n field-led notes (from #1120)
  32. filed for review #1179: verify_plan check for SKILL.md-prose pin tests (from #1134; 3rd recurrence)
  33. filed for review #1180: wf-fix dedup predicate blind to daily-fix: titles (from #1102)
  34. filed for review #1181: lint fix_sha in code-fix relaunch notes + D3 anchor pin (from #1081)
  35. filed for review #1182: memoize transcript resolution across watcher passes (from #1127)
  36. filed for review #1183: relocate live pods_ephemeral.json out of the git tree (from #821)
  37. filed for review #1184: fix the ls-tree fail-open in the Step 10d stale-folder guard (from #1047)
  38. filed for review #1185: trigger-dense artifact review guidance for reviewer agents (from #1058)
  39. filed for review #1186: shared HF tree-pagination helper in verify_task_body.py (merged #1016 iterator + checks-25/30 headers-before-URL ordering candidates)
  40. filed for review #1187: widen thread-caps invariant test target globs (from #1146)
  41. filed for review #1188: analyzer re-fold H1 retitle pairs with set-title (from #1110)
  42. filed for review #1189: /daily headless stub-first mechanical lever (from #994)
  43. filed for review #1190: enforce the HF 10k-files-per-dir upload limit (from #748)
  44. filed for review #1191: lint YAML-scalar truncation + poller-marker consumer reachability (from #873)
  45. filed for review #1192: reviewer writable-tempdir outside the worktree (from #853)
  46. filed for review #1193: root rebase/cherry-pick fence sibling to the #1128 merge fence (from #1128)
  47. filed for review #1194: verify_plan fit-family pilot-basis check (from #1060)
  48. filed for review #1195: forward HF_HUB_DISABLE_XET in the lane passthrough allowlists (from #1049)
  49. filed for review #1196: corpus-audit WARN on H1 vs frontmatter title drift (from #1110)
  50. filed for review #1197: background the Step 1d pristine compare (legit ~640–1950 s vs 600 s cap) (from #1129)
  51. filed for review #1198: prefer explicit-PID kills in crash-fix step 2 (from #848)
  52. filed for review #1199: fix/adjudicate exit-site marker misses red on main (merged #1047+#1053 candidates; test verified failing)
  53. filed for review #1200: sync experimenter memory launcher snippet to the landed pid-file contract (from #1070; agent-memory not a /daily write target)
  54. filed for review #1201: PreToolUse guard on bare git pull at the repo root (from #1047)
  55. filed for review #1202: lint bare list_repo_files in new scripts (from #997)
  56. filed for review #1203: anchor the verify_plan c12 N/A escape standalone (from #879)

Filed for review — transcript-mined today (11 tasks):

  1. filed for review #1204: skip Codex composer spawns under a live quota sentinel (~9+ sessions wasted 3–5 composer runs each today)
  2. filed for review #1205: GCE workload verifies its eval-JSON git push landed (retry-then-fail-loud) — #825's 73 eval JSONs survived only via a pre-poweroff git bundle rescue
  3. filed for review #1206: llm-judging rules 9/23 — API transport errors (429/529) are RETRY, never dropped draws (#1090: 2,638 arm-asymmetric stored-529 "drops", caught pre-publication, 2,635 recovered)
  4. filed for review #1207: codify the long-phase heartbeat duty + tick-cron re-arm (4 watcher force-respawns hit ALIVE sessions today: #1092 ×2, #825, #1112)
  5. filed for review #1208: planner authors Row-coverage: with registered paired contrasts (verify_plan c18 bounced #1112 twice in one day)
  6. filed for review #1209: watcher fast-respawns a session dead on a refused FIRST turn (~100 min gap on #1092 tonight)
  7. filed for review #1210: Step 10d re-snapshots the Guard-1 strip on a merge-conflict error (#1128: two recovery merges, ~35 min)
  8. filed for review #1211: choom-protect the Step 10d lint-gate first run (#1143's no-flags leg died unprotected)
  9. filed for review #1212: Step 10d lint gate runs the origin/main lint vintage (two false-blocks on #1112, ~35–45 min churn)
  10. filed for review #1213: verify_plan WARN on a checkpoint ladder without a retention-sized disk bound (#1133's planner deferred; pipeline planner decides)
  11. filed for review #1214: new_worktree.sh branches from origin/main (3 of 6 sessions needed 10–25 min of merge forensics from local-main churn)

Other problems & notes

  • Route 3 — held for you, filed #1215 (destructive/irreversible): daemon-independent orphan-wrapper sweep — every watcher reaper pass is daemon-gated today, so daemonless zombie wrappers persist; a /proc-based killer is a destructive judgment call (from #818).
  • Route 3 — held for you, filed #1216 (false-fire risk on healthy runs): arm the #864 zombie namespace veto in poll_pipeline — needs discriminator redesign + a live-pod gate re-run before default-ON (from #864).
  • Route 3 — held for you, filed #1217 (architectural per its own plan §13): digest-only bank-read guard arm for real-world-corpus raw_completions — a blocking-behavior hook change (from #1102).
  • Route 3 — held for you, filed #1218 (genuine design fork): gate spawn_session dispatch paths on the auth-outage sentinel — #1027's reviewed plan left them ungated BY DESIGN (from #1027).
  • Still needs you (from yesterday, unchanged): #1140 Codex quota — pay/upgrade vs ride out Claude-only review until 2026-08-06 (all 5 doubled sites single-family today); #1141 HF canonical-repo 100k-file purge strategy; 91 tasks at awaiting_promotion.
  • Live watch item (#1092): P6 GCE fit-grid pilot at ~5.9 h elapsed with a §9 whole-grid projection of ~18–27 h against a 30 h --max-run-duration fence (launched 23:40Z) — if the pilot gate's projection busts the fence, artifacts must be persisted before the DELETE. The session posted the watch item itself; tomorrow's triage should check it first thing.
  • Retraction re-check: 109 marker-mined candidates scanned by the 6 triage subagents against their source tasks' post-park events — 0 suppressed (no retractions found); transcript-mined filings were checked against their own session tails by the miners. 43 candidates were instead suppressed as already-fixed on current main, 2 as dedups (#840, #987), 2 as out-of-scope (src/ experiment code), 4 merged into sibling filings — all 112 rows got epm:workflow-fix-task-filed routed-record markers so the sweep is idempotent.
  • Refusal tax (note-only, mitigations in flight): #1092 lost ~2.3 h across an orchestrator wedge + a first-turn-dead respawn; #1090 took 5 subagent kills (all recovered in-session). The #1127 wedge lane + #1106 sonnet rung landed yesterday; residual gaps filed tonight (#1209).
  • Watcher noise (note-only, fix filed #1207): 4 ALIVE-BUT-STALLED force-respawns hit healthy sessions; one killed a legitimately armed 4 h bg watcher, another directly caused the #1092 first-turn refusal loss.
  • Guards earned their keep (note-only): piped-git hook blocked 11+ attempts fleet-wide (all bare re-runs succeeded); repo-root branch guard, post-merge dup-folder guard, Step 10d guard 3 (38 foreign paths on #1142; stranded-src catch on #1090 → #1144 filed AND merged same day), upload-verifier (#825 push rescue) all fired correctly; foreground-sleep guard down to 2 blocks fleet-wide (from 15+ on 07-03).
  • Merge churn (note-only, fixes filed #1210/#1212/#1214): rebase-merge refused on 4 PRs (documented #1041 squash fallback used); tasks/ conflicts resolved correctly everywhere; no bad merge landed.
  • One-off slips (note-only, all self-recovered <60 s): TaskOutput(task_id="placeholder") (#1146), oversized whole-file Read (#1128), decorative header pasted into a Bash command (#1136), argparse positional slip (#1135), Edit-from-memory anchor miss (#1134), worktree-cd relative-path cp (#1127), heredoc quoting ×2.
  • #1092 experiment-code incidents (note-only, fixed in-session + gotchas filings already routed): vLLM H100 IMA mass failure (35 shards → 1 after eager-mode mitigation), 83-shard dependency-DAG failure, one over-budget conversation killing launch 8 at 99.9% (phase-scoped completion preserved ~1.3 h GPU), P6 GCE staging crash (folder-entry download + unscoped list_repo_files, fixed in one cycle, ~3 min billed).
  • Step C volume flag: tonight's first sweep run filed 60 review tasks + 4 holds in one night (the #1132 plan under-estimated the backlog ~20×). The infra queue will drain at the 5-session cap over days via the watcher backstop. If this volume is unwelcome, bulk-archive candidates are the s*-slugged tasks #1148–#1203 — each carries its full provenance for later revival.

Living-docs drift

  • living_docs.py check FAILs with 199 findings (drift hash 078285c14444): ~75 dead legacy anchors, ~54 completed clean-results absent from any question's evidence, dozens of relates_to ⇄ evidence mismatches (incl. #722/#763/#494–#501), and the stale spec-role-header State trailer.
  • drift proposal already open (logged 2026-07-07, same hash 078285c14444); not re-proposing. The standing proposal remains: re-synthesize open_questions.md (living-docs updater/backfill) — needs your ok; not auto-applied.
  • Parked-proposal re-consideration: no results were promoted today and no questions changed, so no parked living-docs rejection or parked-redundant follow-up became a re-proposal candidate.

My thoughts

Highlighted results

  • no results promoted today