Daily — 2026-08-03 (22 infra fixes landed, GCP backend removed, 0 results promoted)

What happened

• 22 tasks reached epm:done — almost the whole 08-02 daily-fix batch landing at once (#1962–#1978 daily-fix, #1980/#1982 workflow-fix), plus #2032, #2050 (pgrep -f under-match), #2051 (pod bootstrap full-clones 10.5 GB) and #2057 (re-arm the bg poll chain first on every wake). 25 epm:merged, 29 epm:code-review rounds, 24 test-verdicts. Zero blocked transitions.
• #2028 landed: GCP provisioning is now disabled fleet-wide (GPU and CPU, per your "remove GCP fully" directive). The auto chain is fellows → nibi/fir/mila with RunPod as the terminal rung; cpu-bigmem now maps to RunPod cpu5m-16-128. Every eps-issue-* GCE instance reads TERMINATED. #2059 was filed the same day to consider going further (RunPod primary, fellows next).
• 28 tasks created — 16 daily-fix items from last night's sweep, 10 workflow-fixes that sessions filed on themselves during the day (#2049–#2060: duplicate-named pods, keep-running shielding idle pods, lane_suffix in SLURM job names, the no-progress tick wedge, bootstrap-failed pods billing), and 2 new experiments: #2054 (6k-row scaffold-and-splice lattice: framing × character × condition) and #2061 (does post-training make SAE features more predictable from context?).
• 0 results promoted; 1 parked at awaiting_promotion — #1336 (the Llama Tülu ladder's DPO→RLVR step needs at most a rotation of the linear map). The awaiting_promotion queue now stands at 126.
• Live experiment lines: #1739 (evil-OOD arm-fill rounds — 128 markers, the day's busiest), #1482 (SAE Shapley / K-sweep), #1345 (character-capture pivot), #1689, #1947 (localization panel — two live RunPod boxes, 8×H200 + 4×H200), #1895. Cross-family review redundancy was OFF all day: the Codex quota sentinel is live until 2026-08-06, so every doubled review site ran single-Claude under the #1204 no-show fallback.

Applied workflow improvements

1. Target: .claude/skills/issue-tick/SKILL.md — what: pin the canonical worktree-safe REPO_ROOT derivation — commit: 24163612d0 Why: the Step-0 tick recipe said to run "$REPO_ROOT"/scripts/tick_triage.py from a worktree but never said how to derive REPO_ROOT. The #1739 worktree session improvised git rev-parse --show-superproject-working-tree, which in a plain (non-submodule) worktree exits 0 with empty output — so the || fallback never fires and the tick invoked /scripts/tick_triage.py, dying No such file or directory (2026-08-04T01:14:21Z). Verified: reproduced from the live issue-1739 worktree — superproject form rc=0 len=0, the || chain yields '', canonical form resolves scripts/tick_triage.py. Canonical idiom already used at CLAUDE.md:124 and 6 sites in .claude/skills/issue/SKILL.md. Applied edit:

   + Derive `REPO_ROOT` with the canonical worktree-safe form ... — never improvise it:
   + REPO_ROOT=$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")
   + `git rev-parse --show-superproject-working-tree` is NOT a substitute: in a
   + plain (non-submodule) worktree it exits 0 with EMPTY output ...
   

2. Target: .claude/rules/gotchas.md — what: two probed tool traps (RunPod MCP team-scoping; destructive-verb bigram in commit-message prose) — commit: 3311808aec Why (a): the mcp__runpod__* tools send no X-Team-Id, so they see a different personal-account inventory. Probed directly tonight: MCP list-pods --computeType GPU returned exactly 2 EXITED pods (pod-1112, pod-779) and none of the three pods pod.py list-ephemeral reported that same minute — including two RUNNING H200 boxes. An idle/spend audit run through the MCP would have read "nothing live" while 12 GPUs billed. Also mcp__runpod__delete-pod → 500 delete pod: Unauthorized on a team pod (#1482, 18:54:23Z). Why (b): a commit whose message prose carries the two-token destructive-verb bigram trips guard_repo_root_branch.sh — the arm regex at L1698 anchors anywhere in the argv, and the -m payload is on that argv. #1482 hit it at 23:22:18Z writing up an incident. Re-confirmed live while writing the entry: my own verification call carried the bigram in a quoted test string and was refused, and the deny killed the whole compound. This commit itself used the prescribed -F <file> form. Applied edit: two bullets appended to # Gotchas (RULE + incident citations, matching the file's register).

Lint gate after both: --check-references PASS, --check-asks PASS, --check-lessons-index PASS (with a pre-existing WARN — see notes).

3. filed for review #2062: lean-context subagent twins — the autocompact-thrash ladder has no rung past orchestrator self-compose (~29 thrash deaths across six sessions in one evening; micro-scoped default-model respawn, the only prescribed remedy, failed repeatedly — 3 deaths on an 11-line diff, 4 at a 4-tool-use signature).
4. filed for review #2063: judge max_tokens pins sit below the rule-23 floor they cite — 15 pins in scripts/ still at 300/400/600 after the 2026-08-02 raise to 1024/2048 (12ce0a8225), most citing rule 23 as their justification. Below-floor budgets truncation-censor draws arm-asymmetrically. (wf_fix: false — experiment scripts.)
5. filed for review #2064: registry drift at an allocated id breaks task.py new fleet-wide — tasks/proposed/2052 existed unregistered, the allocator re-issued 2052, and every filer in the fleet failed on a bare FileExistsError (0 handlers on that path).
6. filed for review #2065: flag dirty-tree provenance in result-JSON reproducibility metadata — this routes a candidate session 201e2896 wrote up and then parked for your greenlight instead of filing, which workflow-fix-on-bug.md explicitly bans. Both #1482 Shapley runs consumed a script modified +369/−40, so committed figures record a git_commit that does not contain the logic that produced them. (Already dispatched — at planning.)
7. filed for review #2066: guard_root_code_commit.sh is cwd-blind — worktree commits blocked by foreign files in the shared root index; ≥7 commits blocked across 5 unrelated sessions, each costing 2–4 turns. Cross-references open sibling #2046 (same file, pathspec case).
8. filed for review #2067: a cross-session lane pivot must resolve the owning session before provisioning — a #1345 pivot by a non-owning session left pod-1345-charcapa (8×H200) idle 74+ min at 0% GPU (~$54) while the owner waited on the abandoned fellows queue.
9. filed for review #2068: forensics — a concurrent repo-root working-tree revert scoped to eval_results/ silently destroyed a completed 48-minute ridge run's JSON (#1482), despite the guard that refuses exactly that being verified armed and firing today. Evasion path unknown; that is the task.
10. filed for review #2069: on-policy datagen must persist judge-rejected generations — #1689 ships only kept rows, so inspecting rejects required a full regeneration, contradicting persist-by-default (rollout text is never discardable) and destroying the rules 9/23 audit trail.
11. filed (needs-human) #2070: the 61-deep auto-dispatchable infra queue is now the binding constraint — see notes.

Dispatch state of tonight's filings: #2065 dispatched (at planning); #2062/#2063/#2064/#2066/#2067/#2068/#2069 are dispatch pending at proposed — the filer's spawn no-op'd on the session-dispatch stagger / cap, so the watcher's proposed_infra_sweep backstop owns them (see logs/daily/filings-2026-08-03/filed.jsonl). #2070 is deliberately --no-dispatch (route 3, needs-human). That 7-of-9 deferral is itself the condition #2070 asks you to decide about.

Other problems & notes

• Route-3 held (needs you) — #2070, infra dispatch backlog. 93 proposed tasks, 87 kind: infra, 61 auto-dispatchable behind a 5-session cap with 36 live sessions. Concrete cost: #2022 (a one-line tests/sparse_cones.txt fix whose absence reds fresh sparse worktrees' Step 9c at collection) was filed 2026-08-02T17:55:48Z and has never dispatched — one epm:created event, ~30h — and the same bug re-bit twice since. #2006 (idle-pod teardown) likewise sits proposed while that leak recurred again tonight. Not a bug (#1853 already fixed the sweep mechanism and is closed) — a resource/priority call: raise the cap, prune, priority-order by severity, or tighten /daily's route-2 bar. Immediate ask: dispatch #2022 by hand.
• Route-3 held (needs you) — already tracked in #1686 (shared: draft, root, scripts, shared, unrelated, untracked; anchor: root): six untracked draft scripts squatting at the shared repo root (issue1482_* ×4, issue1895_overlap_ksweep_fig.py, issue1768_result2_ladder.py). Duplicate filing suppressed by the #1483 overlap dedup; I posted a freshness note on #1686 instead, recording that the set has grown, that it now blocks commits fleet-wide (not just degrading the Step-9c oracle as originally filed), and that issue1482_shapley_blocks.py should be committed rather than deleted because its uncommitted diff is what produced #1482's figures. Deleting untracked files is not git revert-able, so /daily may not self-apply.
• Step C parked-candidate sweep: 3 candidates enumerated, all dispositioned. (a) #1965's tests/sparse_cones.txt park → deduped against open #2022, routed-record posted on #1965 with the ~30h non-dispatch escalation. (b) #1968's park targets ~/.claude/skills/converge/SKILL.md — user-global, out of /daily's surface by definition (global files are /memory-sleep's remit, and a route-2 /issue worktree merge cannot land a file outside the repo); routed-record posted on #1968 with the misclassification note so the emitting pattern gets corrected. The substance is sound and small — mirror the #1968 storm-liveness hold into that skill's L682 bare-429-retry instruction — and belongs to the next /memory-sleep. (c) #1771's architectural-greenlight mechanization park → deferred again, 7th consecutive night: #1771 itself is still plan_pending rewriting that exact gate, so the "target under active change" deferral still holds. No suppressing routed-record posted (that would bury a genuinely pending re-consideration). 1 benign skipped row (#1333 json-decode, relevant_kind: false); unmatched_record_fps 326 refs — the standing #1703 advisory backlog, unchanged in character (326 yesterday too).
• Retraction re-check: 3 marker-mined parked candidates scanned (fresh events-tail read per park), 0 retracted upstream. Transcript-mined filings inherited the miners' full-session tail reads; two premises were re-scoped before filing — the guard-cwd finding was narrowed once #2046 was read (pathspec case ≠ cwd-blind case), and the infra-backlog item was reframed after reading #1853 (mechanism already fixed; the residue is a resource call).
• Landed-fix / sibling suspects overridden after eyeballing (2, both false positives of the #1735 generic-vocabulary class): guard-root-commit-cwd-blind vs commit 638093ec4f (that commit is the #1857 cert-retry settle-and-re-hash pass — it changes which content gets hashed, not the Layer-2 root-index read; shared tokens were 1620,guard,hook); persist-rejected-judge-draws vs #2020 (the on-policy yield-floor close-miss escalation — 0 mentions of "reject" or "persist"; shared token on-policy + same target file).
• #1739 lost ~7.8h to a refusal-then-context wedge and the watcher did not respawn it. Session b0db8707: 17 isApiErrorMessage rows 10:27–18:16Z — 8 usage-policy refusals ("violative cyber content"), 8 "Prompt is too long", 1× 529 — every /issue-tick wake dying in an api-error row, recovery only via harness autocompact at 18:16. A second #1739 session (9e7a6e7e) died the same way at 00:47:19Z when a STALE-REDRIVE reloaded the 44K-token SKILL.md. The prompt-wedge lane (EPM_TICK_WEDGE_MIN_FAILED_TURNS, #1127/#1453) should have force-respawned after ≥3 consecutive failed wake turns; 16 accumulated. Not filed — #2058 (approved, no-progress tick wedge) is landing in this exact area and may or may not cover the api-error-wake path; worth confirming when it merges rather than filing a possible duplicate into a 61-deep queue. Contributing cause on the refusal side: #1739's own round labels ("evil-ood-spread-round") ride into every tick-visible surface, the trigger-dense-by-reference discipline this line keeps re-learning.
• Two money leaks that you spotted before the fleet did. (a) Two orphan bare-named pod-1739 duplicates idled 29.2h and 15.6h at ~$4/hr (~$187, 0% GPU, empty logs) — the issue-wide keep-running tag shielded them from every watcher auto-stop and the provisioning call appears in no surviving record; you asked "are you running something here". (b) pod-1947-loc (4×H100) sat RUNNING after a failed bootstrap ~25 min until you asked "is this bugging" then "can you fix workflow so this doesn't happen again" — filed same-minute as #2060. Both already routed (#2049 duplicate-named pods, #2052 keep-running shielding idle pods, #2060 bootstrap-failed teardown), all three still proposed.
• Repeated user re-steers worth naming, all recovered in-session. #1482: the full-dictionary width had to be re-pinned 4 times after the first directive ("ONLY DO THE FULL 128512" → later sub-analyses defaulted back to the 16k panel). #1739: a build-unit script never set mapfit, silently skipping arms 6/7/8/13 — you caught it and asked for the fix and a guard against recurrence (landed 702e0c3535, 26/26 mapped tests); separately my arm-count and coverage claims were wrong in both directions twice, and two rounds of "never ran OOD" claims were refuted by artifacts in sibling directories. #1895: cross-space R² presented as a head-to-head before target-matching, 4 pushbacks, and you supplied the fix. #1491: you said n=10k and the plan was unilaterally switched to n=25k without a confirm. #1345: a writeup ignored the literal skeleton you handed it. Wall-time was never volunteered at dispatch (6 "how long will this take" asks in one session) and parallelization had to be requested 3 times.
• Guard-block compliance ledger (hooks all worked; these are wasted turns, not gaps). Piped git push/commit/gh pr create blocked ~17 times across ~12 sessions; foreground sleep chains ~9; uncertified root-code commits ~13; repo-root branch/destructive ops ~5; bare | python -c 1. Every one recovered on the next call. Three guard false-positives are worth a fix if they recur: an SSH-remote pod-side git checkout/reset --hard blocked by argv text-matching (#1345), a worktree-cwd git pull --rebase blocked the same way (#1491), and the commit-message bigram now documented in gotchas.
• .claude/rules/LESSONS.md is at 7564/8000 bytes — inside the WARN band (>7200). The next rule addition may FAIL the lessons-index check; largest row is gotchas (1170 B). Not caused by tonight's edits (I did not touch LESSONS.md — the two gotchas bullets went into the rule file, not the index). Needs a slim pass or a deliberate cap decision soon.
• pod-1739 has been STOPPED for 2 days while #1739 is at followups_running with the keep-running tag set — a STOPPED volume is not durable (#1112), so anything only on that volume is at risk. The stale-pod audit reports it as kept-exited (tag-exempted) rather than reaping it.
• Smaller recurrences, each recovered: TaskOutput keyed on a teammate name instead of the spawn task-id (3 sessions — documented anti-pattern); Skill("happy:change_title") for what is an MCP tool; unscoped HF full-tree listings wedging twice before switching to list_repo_tree; a Step-9c pytest launched as bg-Bash with a 5700s timeout silently clamped to the 600s cap and SIGTERM'd at 92%; a Step 10d lint-gate background job vanishing with no verdict file (Monitor waited 30 min on silence — an EXIT-trap verdict write would fix it); 4 duplicate /issue sessions correctly detect-and-exited at the Step 0 guard; one /issue 2053 spawn against an already-archived task.

Living-docs drift

• living_docs.py check: FAIL — 199 findings (legacy relates_to anchors, evidence-list mismatches, completed-results-missing-from-evidence). Normalized hash 078285c14444 — identical to the 07-07/07-22..07-28/08-01/08-02 rows and materially the same drift as the open 06-29/07-04/07-06/07-07 re-synthesis proposals.
• drift proposal already open (logged 2026-06-29 → 2026-08-02); not re-proposing. Event row appended to .claude/cache/nightly-consolidation-events.jsonl.
• Parked-proposal pass (Step B): #713's epm:living-docs-update-rejected remains un-reconciled with no new context bearing on it (skipped, no spam); no on_hold task carries epm:followup-parked-redundant provenance — no revival candidates.
• Title-sync sweep (Step D): title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-25.md). Read-only sweep exited 0.

My thoughts

Highlighted results

• no results promoted today