EPS
← All docs·Activity

Daily — 2026-07-21 (12 overnight fixes merged;

updated: 2026-09-03

What happened

  • All 12 of last night's daily-filed fix tasks (#1574–#1585) ran their autonomous /issue pipelines end-to-end and merged to main (PRs #1349–#1360): the GCP/SLURM log-excerpt structural digest (#1574), the expensive-band cap-park note (#1575), the WARN-only inline-interpreter lint arm (#1576), the trigger-dense Read-bounding hook (#1577, live-probed BLOCKED/OK as designed), the completed-unmerged resume-table row (#1578), the step9c .sh selector arms (#1579), the fingerprint tag/body reconcile (#1580), the edit-success gate (#1581), the wedged-owner pod escalation arm (#1582), the teammate-idle CLAUDE.md rule (#1583), the merge-scoped gitleaks hook (#1584 — merge-stage scan 0.246 s vs the ~50 min #1345 baseline), and the inline-lint-gate false-positive fix (#1585, filed mid-day from the #1112 session and completed the same day).
  • #1345 recovered and parked: the stranded PR #1347 (prior session auto-stopped after committing its conflict resolution 064f766ee0 but before pushing) was squash-merged fd23b57c60 at 07:05Z; task at awaiting_promotion.
  • #1112 rankem inline round — a full compute odyssey: smoke 4 green, then three crash/fix cycles (full-FT completion-masking BPE-seam assert → fixed via per-segment token-ID concatenation + regression test 6321bc5e4; a MooseFS stale-read serving the pre-fix script; HF private-LFS 403 billing wall at the first checkpoint-overflow upload) → parked ~22 h on the HF wall → the STOPPED pod then vanished provider-side (volume + resume state lost) → fresh RunPod provision blocked by the account hourly cap → user directed "switch to GCP" → flex-start hit the #1116 queue-vanish class (RunPod failover also capped → terminal) → manual GCP on-demand pin hit ZONE_RESOURCE_POOL_EXHAUSTED → 12-attempt/10-min retry loop armed at day end. The user cleared the HF billing wall in the interim (private-LFS upload probe OK).
  • #1586 filed + spawned on user directive (matched-install LoRA vs full-FT confirmation seeds); in planning at day end, with a wider-parallelism steer delivered via epm:progress.
  • Evening chat block: two parallel sessions independently synthesized the predictor line ("does the context→answer map help predict behavior pre-generation?" — answer: predicts, but doesn't yet beat simpler baselines); a #1310 story-character-mapping Q&A surfaced that cross-character map similarity was never measured (user: "run this now inline."); a #1092 draft-writeup review caught the draft's Result 6 claim contradicted by the recorded data.
  • Infrastructure weather: two Anthropic 529 overload storms (~07:05–07:30Z and ~11:15–11:26Z) killed ~9 subagent turns across 5 sessions — all recovered via the resume/re-spawn ladder, zero work lost. Codex quota outage continues (sentinel until 2026-08-06): every review site fleet-wide ran single-Claude; the #1204 pre-spawn skip worked every time (zero wasted composer spawns).

Applied workflow improvements

  1. Target: CLAUDE.md — what: teammate-channel scope note: SendMessage reaches only Agent-tool subagents; an independent Happy session is NOT SendMessage-addressable — steer it via a durable epm:progress marker — commit: f985fec873 Why: session 62e315d1 (2026-07-22T05:58Z) tried to SendMessage a spawned /issue 1586 Happy session to deliver the user's wider-parallelism steer; the channel doesn't reach independent sessions (2 wasted tool calls before self-correcting to the marker). Applied edit: 
    + Channel scope: SendMessage reaches only SUBAGENTS spawned via the Agent tool — an independent Happy session (`spawn_session.py spawn-issue`/`spawn-pm`/`spawn-campaign`) is NOT SendMessage-addressable; steer it via a durable task marker (`epm:progress` on its issue) instead (#1586, 2026-07-21).
  2. Target: .claude/rules/gotchas.md — what: new bullet: single-quote python -c payloads containing $(/backticks/$ — bash command-substitutes them and silently corrupts the payload — commit: 385744dd2e Why: session aa8882d9 (#1581, 09:34:26Z): a double-quoted one-liner's $(.*?) regex was executed by bash (/bin/bash: line 8: ..: command not found) and the "extract Assumptions section" command printed the wrong section — self-masking, no crash. Applied edit: 
    + - **A double-quoted `python -c` payload lets bash execute `$(`…`)`-shaped text inside it as command substitution — single-quote the payload (or use a quoted-delimiter heredoc).** [...] (Incident #1581 session, 2026-07-21.)
  3. filed for review #1587: dispatch-time auto-adoption of a durable trigger-dense task tag at /issue SKILL.md Step 6d.2 (recognition heuristic + durability pin + Step-9c selector registration) — routed from the #1574 recursion-guard park.
  4. filed for review #1588: resolve + document one reading of the SKILL.md autonomous-block step-1 reconcile scope (four cited spans in tension; #1575's fix is correct under either reading) — routed from the #1575 park.
  5. filed for review #1589: a step9c selector-module diff selects ZERO tests (test_step9c_baseline.py references the selector by symbol only; empirically re-confirmed at filing) — routed from the #1579 park.
  6. filed for review #1590: tests/conftest.py:288 hasattr-guarded asw stub — test_no_hasattr_asw_stub_guards is RED on pristine main right now (re-reproduced at filing); one-line fix — routed from the #1582 park.
  7. filed for review #1591: widen the piped-git guard to hook-running git commit | head pipes (SIGPIPE kills the commit mid-hook; target corrected to .claude/hooks/guard_piped_git_push.sh) — routed from the #1584 park.
  8. filed for review #1592: planner must self-count grep -c-style acceptance-criterion patterns against its own verbatim insert text — the same defect cost a revise round in BOTH #1581 and #1583 this morning.
  9. filed for review #1593: self-derive the WORKFLOW_INVARIANT count pin — the hardcoded integer caused a real cross-session merge conflict today (#1575 vs #1584, 42→43 union, ~15 min).
  10. filed for review #1594: MooseFS fix-engaged verification needs a pod-side CONTENT read (sha256/probe-execution) — git HEAD/ancestry passed while FUSE served stale pre-fix bytes (#1112 attempt-2 crash, ~$3-4).
  11. filed for review #1595: persist done-JSONs/resume sentinels to HF before any multi-hour pod.py stop — pod-1112's stopped volume vanished provider-side despite keep-running + the 7-day idle window.
  12. filed for review #1596: the #1116 queue-vanish failover should retry GCP on-demand rungs when RunPod refuses for CAPACITY (not crash) reasons — #1112 needed a ~20 min manual --provisioning-model STANDARD recovery.
  13. filed for review #1597: extend the pod-side relaunch log-rotation duty to inner append-mode per-cell logs (a crash-signature grep false-fired "ASSERT-FAILED-AGAIN" on the previous crash's lines).
  14. filed for review #1598: fan-out teammates must make the report SendMessage their FINAL turn action — 6 idle-without-report nudges across 3 sessions in one morning.
  15. filed for review #1599: sweep parked-candidate subsumption should compare against fix-task MERGE time, not creation time — tonight's own Step C enumerated the stale #1577 candidate 64 s after #1579's fix merged; only a manual dedup prevented a duplicate pipeline.

All 13 route-2 filings have terminal filed rows in logs/daily/filings-2026-07-21/filed.jsonl; spawned sessions are live (e.g. #1596 in clarifier at 06:49Z); any cap-deferred spawn is covered by the watcher proposed_infra_sweep backstop.

Other problems & notes

  • Red pin test on main (also filed as #1590, flagged here for visibility): every full-suite Step 9c run currently carries the pre-existing test_no_hasattr_asw_stub_guards failure until #1590 lands — sessions must attribute around it.
  • #1577→#1579 sibling-blindness (the near-miss behind #1599): #1577's session parked a candidate for the exact selector gap #1579's session merged 64 seconds later; tonight's sweep enumerated it as a "genuine re-raise" (creation-time rule) — deduped manually against the landed fix (f0770307ce), routed-record posted on #1577. The #1479/#1476 duplicate-pipeline class (~2.3 h) was avoided.
  • Codex quota outage (to ~2026-08-06): re-raised by 4 of 7 miners as a judgment call — already tracked in #1472 (blocked daily-held); no new filing. Cross-family review diversity absent fleet-wide for ~2.5 more weeks.
  • #1112 notes: the HF private-LFS billing wall (403) that parked the round ~22 h was cleared by Thomas in the interim (upload probe OK); the vanished stopped pod is covered by #1595; the RunPod account hourly cap + the genuine GCP stockout are external capacity events (retry loop armed; Thomas offered a cap override as escape).
  • Duplicated fan-out (user-initiated): the same predictor question was asked in two parallel Happy sessions 27 s apart — ~6 duplicated subagent sweeps. Nothing concrete to dispatch (deliberate A/B is plausible); FYI: one summary session per question suffices.
  • Synthesis clarity: in BOTH predictor sessions Thomas had to follow up "so it doesn't help us to predict behavior pre-generation?" before getting the crisp bottom line — the SOUL.md lead-with-the-answer discipline; no surface change filed.
  • Implementer refusal-kill on #1574's report turn (98 tool calls done, all work durable; rung-(b) respawn recovered in ~11 min) — the kill was at the subagent's own OUTPUT turn on a trigger-dense task; no brief discipline can prevent that shape; note-only.
  • 529 overload storms: ~9 subagent/turn kills across #1574/#1575/#1580/#1582/#1583/#1584/#1585 sessions + one /issue-tick; all recovered (SendMessage resume, pacing waits, re-spawns); one resumed planner returned HTML-escaped output (known quirk, canonical extraction recipe used); note-only.
  • #1584 Step 10d merge conflict on the shared WORKFLOW_INVARIANT count pin (merged on attempt 3, ~15 min, fresh-context conflict subagent) — the root cause is filed as #1593.
  • Under-scoped plans/filings caught in-pipeline: #1577's plan v1 asserted the selector maps .sh (the opposite was true); #1578's plan had a wrong grep count + stale line numbers; #1581's plan v1 falsely claimed --check-asks is in the workflow_lint no-flags dispatch (verify_plan c37 caught it). All corrected by the fact-checker gate; note-only.
  • Guard-hook blocks working as designed (no filings): piped-git push block ×1 (yesterday's /daily session — deviated from its own recipe, hook caught it), root-code-commit certification block ×1, foreground sleep-poll blocks ×2 (#1112, #779 scoping), guard_trigger_dense_read probe block ×1 (deliberate test). Plus: one index.lock collision (retry-once OK), one TaskOutput called with a teammate name (usage error), one cd into a nonexistent relative path, one malformed HF list_repo_tree path (retried OK), 3 premature reads of empty bg-output files + 4 Bash timeouts auto-moved to background (slow MooseFS/SSH), one stale relaunch wrapper killed after its work was verified live.
  • verify_plan WARN c38_exit0_repo_wide_baseline acknowledged + carried on #1578 (working as designed).
  • #1345 stranded-merge root cause confirmed: the prior session was auto-stopped after its conflict subagent committed but before pushing — exactly the class the landed #1564 completed-unmerged watcher pass now flags; no new filing.
  • Daily-stub push race (this run): first push rejected (concurrent committer), recovered by sync_repo_root.py (state=synced, push landed).
  • Title-sync drift sweep (Step D): title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-11.md for the verbatim rows).
  • Retraction re-check: 6 marker-mined candidates scanned, 0 retractions found (1 deduped against a landed fix — the #1577 candidate); transcript-mined items were tail-checked by the miners through end-of-transcript.

Living-docs drift

  • living_docs.py check: FAIL (RC=1) — 200 finding lines (hash 114f706ff987, churns on title/count tokens), materially the SAME drift as the still-open re-synthesis proposals of 06-29 (1736fca0804f), 07-04 (d2a80db46b3b), 07-06 (b009cef9ecd9), 07-07: ~75 dead legacy anchors, ~54 completed clean-results in no question's evidence, the relates_to⇄evidence mismatches, and the stale spec-role-header State trailer.
  • drift proposal already open (logged 2026-06-29, extended 07-04/07-06/07-07/07-19/07-20); not re-proposing — event row appended to .claude/cache/nightly-consolidation-events.jsonl.
  • Parked-proposal re-consideration (Step B): no epm:living-docs-update-rejected markers awaiting reconciliation and no on_hold parked-redundant follow-ups — nothing to re-surface.

My thoughts

Highlighted results

  • no results promoted today