EPS
← All docs·Activity

Daily — 2026-08-04 (autocompact-thrash storm day; root cause found + fixed in-window; recovered by the 08-05 nightly)

updated: 2026-09-03

What happened

  • RECOVERY NOTE: the 08-04 nightly /daily died at 2026-08-05T07:03Z on repeated API 529 Overloaded errors ~36 min in (stub + miner brief only). This file was recovered via the SKILL.md Edit-in-place husk branch by the 2026-08-05 nightly session, which re-mined the full 08-04 window (7 miners, 78 transcripts, ~29k rows).
  • The day's dominant event was a fleet-wide autocompact-thrash storm: ~46+ subagent deaths across ≥16 sessions (planner/critic/implementer/reviewer, default AND lean twins), plus orchestrator-level thrash loops (one session: 35 compacts/11 errors; #2004's: 44 continuation restarts). Tasks #1217, #1336, #1491, #1771, #1983, #1994, #1995, #1996, #2001, #2005, #2054, #2061 all blocked or degraded. Root cause was found and fixed in-window, largely driven by Thomas: Happy's vendored Agent SDK (2.1.128) resolved the opus[1m] settings pin to claude-opus-4-7 and stripped [1m] from subagents, compounded by the tengu_amber_redwood reduced-window enrollment — fixes: CLAUDE_CODE_AUTO_COMPACT_WINDOW=1000000 env (14:45Z), lean-twin agents (#2062, 12:42Z), Happy daemon patch (~21:31Z), default spawn model → fable (21:23Z, commit 652d9a7bce), #2073 filed + landed. The 08-05 04:46Z probe table verified the fix held (fable sessions 618–645k tokens, zero compactions).
  • The styfeng pod incident: Thomas relayed that another fellow's pod was cancelled — diagnosis confirmed the EPS pod-audit cron terminated styfeng-8xH200 on 07-31 (substring ownership inference self-poisoned), with a 13-day near-miss on fellows-cluster nodes. The kill-approval interlock (EPS_ALLOW_POD_TERMINATE/EPS_ALLOW_COMPUTE_KILL) landed in-session; the "nothing gets shut down on its own" directive was issued and codified.
  • The architectural-greenlight gate was removed on Thomas's order ("remove the architectural-greenlight gate and get all these 65 proposed infra tasks done") — the 65-task infra drain then started (cap raised 5→8, later tuned), with the Happy daemon outage (21:13Z–04:30Z, ~3h17m) freezing dispatch overnight.
  • #1491 ran its inline scale-ladder day: a P0 manifest crash (wrong-domain sha pins, the #1776 class re-introduced — fixed with fails-pre-fix tests + #2079 filed) and a plan-level Gate-1 null-predicate defect that killed two rungs before an advisory floor held. #1947 burned ~$500 on two ~10 h idle pods (an unowned HOLD gate; a bootstrapped-never-launched pod). #1739/#1482 ran long user-directed analysis days with heavy Thomas iteration (blinded reads, figure coverage, judge-refusal instrument fix).
  • 0 clean-results were promoted on 2026-08-04.

Applied workflow improvements

The recovery sweep routed everything through the 08-05 nightly's classifier. Route-1 items derived from 08-04 evidence were applied there (see logs/daily/2026-08-05.md Applied items 1–2 — the gotchas.md load_dotenv-stdin trap is the #2054 08-04 firing). Route-2 filings from this recovery sweep (all tagged daily-auto-filed, filed 2026-08-06):

  1. filed for review #2139: tick_triage counts auto-compact continuation rows as HUMAN activity — a false HEALTHY per compaction masked the #1336 (~5.4 h) and #2054 wedges (probed: no isCompactSummary handling).
  2. filed for review #2135: pod HOLD gates + post-provision launches must arm a re-drive / confirm a live workload — the two #1947 idle pods (~$500).
  3. filed for review #2136: ensemble_verdicts_present returns a stale prior-round PASS (no round token) — near-miss skipped review on #1336.
  4. filed for review #2137: pod_audit ownership by substring self-poisons (styfeng + 13-day near-miss) — structured-evidence inference + the creation-age clock bug; the interlock stays the backstop.
  5. filed for review #2138: reap-husks comparator reads a symlinked plan.md as unique content — the #2051 husk held a dispatch slot during the drain.
  6. filed for review #2140: watcher daemon-liveness pass — the 3h17m Happy daemon outage froze all autonomous dispatch with no escalation.
  7. filed for review #2141: subfloor auto-reclaim never fired through ~7 h below its arm point + watcher logs named by local date (UTC greps read nothing).
  8. filed for review #2142: dispatch-lease races — four duplicate /issue spawns (9–59 s gaps) + a cap overshoot (occupied=6, cap=5).
  9. filed for review #2143: capture the bare-API blinded-read recipe as a rule — three leakage corrections on #1482 before the design converged.
  10. filed for review #2144: pre-registered gates on NULL statistics need a measured calibration basis — the #1491 Gate-1 predicate killed two rungs.
  11. filed for review #2146: the inline-override duty list misses the tick-backstop-cron arming — surfaced only by Thomas's "did you go through the entire issue skill" challenge.
  12. filed for review #2145: dispatch_issue.py launch lacks --name-suffix passthrough + a --gpu-type override — two burned launch attempts, intent abuse for 8×H200.

Corroborating-evidence notes were posted on three of the 08-05 filings rather than double-filing: #2115 (worktree-vs-main permission parity datum for the lint-gate wedge), #2124 (judged-DV completeness gate candidate — the #1739 28.3% rubric-REFUSAL censoring), #2126 (Guard-1 extraction, selector family-binding, Step 6d pre-spawn pod check).

Other problems & notes

  • Still-blocked residue of the thrash storm — needs a PM re-drive now that #2073 landed: #1217, #1983, #1994 remain blocked with thrash-exhausted review ladders (probed at mining time); each is a small reviewable diff. #1983's blocked push notification also failed to deliver ("Remote Control inactive").
  • In-window landed fixes (no filing needed): #2062 lean twins + [epm-inline-fallback] token; #2073 compact-window override + gotchas discriminator; #2058 no-progress respawn lane (its own sessions wedged twice while building it); implementer-lean created (68f7d5e543); kill-approval interlock; DEFAULT_SPAWN_MODEL="fable" (652d9a7bce); bootstrap sparse-cone ssh-payload fix (ded0ccc9bd); #2022 sparse-cone registry fix.
  • Already tracked (deduped, no new filing): cwd-blind root-commit guard → open #2066; lean-twin registry visibility (frozen agent list at session spawn; worktree branches lack the files) → open #2072; gc.log/git-prune wedge → open #2007; kept root-sync stashes → #1736/#1922; teammate stand-down/ownership-release → open #2034.
  • Thomas-corrections (recurrence data on existing rules): "you didn't actually start" (planned instead of executing after an explicit go-ahead, Happy daemon fix); repeated shallow absence claims on #1482 ("search deeply" ×4) + a stale "cannot be run" claim a landed follow-up had fixed; blinded-read leakage ×3 (filed as #2143); figure URLs not volunteered ×2 + dropped plot promise; lane strategy contradicting the standing runpod-first directive (#2059) — root fix #2054 landed 08-05; find-session ask resolved to a 3-week-stale session before recency was considered (wrong session actually resumed, then swapped); inline/parallel dispatch split still typed by Thomas 4×; opaque regime codes (e2p_fc) in a chat table; missing figure bars presented silently (≥3 asks) — the interim-figures N/A — not tested convention is the existing rule.
  • 529 storms (~14:46–15:10Z + evening): killed two orchestrator sessions outright, boot-killed fresh spawns on #1217/#1988/#1990, and killed the nightly /daily itself. Recovery machinery (ticks/watcher/successors) handled most; the boot-death + wedge lanes were the gaps now filed (#2113 auto-backfill; #2115's watcher fast lane).
  • #2054 sat blocked ~10.6 h on unit_b_c_compose_wall with no push to Thomas — he discovered it himself ("why did it stall"); the lean-twin teammate then landed Unit B/C in ~7 min. Watcher gate-push coverage of failure_class: infra blocks on user-origin experiment tasks is worth checking (folded into #2111/#2140 scope at plan time).
  • Step 9c/10d frictions: spec-freshness family sync pulled the step9c selector without its siblings (2 broken gates on #1999, ~70 min — noted on #2126); a self-inflicted WORKFLOW_INVARIANT/manifest mismatch cost one ~30-min gate (#2067); main was red with 4 unledgered test_pod_lifecycle failures (two sessions re-derived the same pre-existing red — the ledger cron #2114 is the relief); select_step9c_tests --map-files misused 3× across two sessions (its #1613 guard could auto-wrap; noted for #2126's planner); step9c_baseline xunit1 + dirty-root ergonomics cost two extra rounds.
  • Guard-hook blocks: ~20+ firings across the window (piped git, uncertified root commits incl. one pathspec-less commit that would have swept a sibling's staged files — the #1894 hazard the hook exists for, repo-root branch/rebase, log-dump cat, sleep chains). All recovered; frequency is the standing compliance signal.
  • task.py post-marker --note backtick mangling (one marker corrupted + correction marker, #1491) — evidence FYI'd into #2121's scope (same target file) via its body's origin trail; address-concern refused 4 in-context concern-ids with an unhelpful error (noted for #2121's planner).
  • Misc probed one-offs: CronDelete called without loading its deferred schema (recipe could name the ToolSearch preload — noted for #2126); /model-adjacent Skill(happy:change_title) stale form fired despite the doc warning; commit-msg file vanished from /tmp mid-run (compose under ~/.claude/tmp/); uv cache lock timeout at 107 concurrent uv processes; earlyoom killed 7 processes; / 92→96 % + eps-data 96 % through the day (the disk crisis that continued into 08-05 — route-3 #2131/#2132 carry the decisions).
  • Retraction re-check: marker-mined candidates for this window were routed through the 08-05 Step C sweep (8 scanned, 2 premise-retracted); the recovery sweep's own filings were composed against the CURRENT tree per the backfill rule (landed fixes deduped above).

Living-docs drift

  • Recovery run — the living-docs passes (drift check, Step B re-consideration, Step C parked-candidate sweep, title-sync) executed once in the 2026-08-05 nightly session that performed this recovery; see logs/daily/2026-08-05.md § Living-docs drift (200 findings, open re-synthesis proposal stands; no re-proposal).

My thoughts

Highlighted results

  • no results promoted today