EPS
← All docs·Activity

Daily — 2026-07-29 (main un-redded; 6 applied, 29 filed)

updated: 2026-09-03

What happened

  • Huge infra day: 62 tasks created (#1785–#1846, mostly the 07-28 nightly's filings) and ~30 daily-fix/workflow-fix pipelines ran end-to-end and merged to main via Step 10d (#1785–#1813 chain + #1829/#1833/#1834/#1838/#1844) — all reviews single-Claude under the fleet-wide Codex quota outage (sentinel live until 2026-08-06).
  • Experiments: #1773 and #1738 landed clean-results (both parked/folding follow-ups); #1769 and #1775 reached awaiting_promotion (each with cheap-band follow-up rounds running); #1776 finished its full workload (pods p0–p5 + judge + analysis) and passed the clean-result gate; #1689 completed its wider-lambda-ceilings round; #1482 folded early-layer + entered a matryoshka-tier round; #1768 (mt-battery, 8×H100) hit a 12–24× p8 sizing deviation and re-widthed; #1739's 3-lane GCP run survived a syc-lane crash (per-slice pinv fix, relaunched with resume).
  • Main was red ~40h from the 07-28 daily's own route-1 commit (c341f3bd59, E501 masked by per-file-ignores) — every session's Step 9c gate re-paid classification; fixed tonight (see Applied #1). The fix task #1823 sat undispatched 22h (dispatch-gap filed as #1853).
  • No clean-results were promoted today.

Applied workflow improvements

  1. Target: scripts/verify_task_body.py — what: re-wrapped the inclusive-cap f-string at L13224 to 100 cols, un-redding test_ruff_policy::test_live_workflow_helpers_clean_under_full_ruleset on main (red ~40h, fleet-wide gate cost) — commit: 7202cfd669 Why: miners A/E/G/J all flagged the live red + undispatched #1823; probed red via the exact test node, green after the edit; inline payload lint gate PASS (cert), string output byte-identical. Applied edit:
    -                f"<{V3_FINDING_PROSE_FAIL_WORDS}; FAIL fires at ≥{V3_FINDING_PROSE_FAIL_WORDS} inclusive)"
    +                f"<{V3_FINDING_PROSE_FAIL_WORDS}; FAIL fires at "
    +                f"≥{V3_FINDING_PROSE_FAIL_WORDS} inclusive)"
    
  2. Target: .claude/rules/llm-judging.md, CLAUDE.md, .claude/rules/experiment-guidelines.md — what: dropped stale hardcoded guideline counts ("23-guideline"/"21 judging guidelines"/"(23 guidelines)") that drift each time a rule lands (#1801's parked candidate; rule 25 made them off-by-two) — commit: d7fb12d9a7 Why: parked candidate on #1801 (fp 884d8dd2165a); verified rule 25 merged + count strings present at the three sites. Applied edit: the full 23-guideline recipe (the 21 judge-scoring guidelines plus… → the full guideline recipe (the judge-scoring guidelines plus… (mirrored at all three sites).
  3. Target: .claude/agents/planner.md, .claude/rules/planner-section-reference.md — what: synced the two c39 doc spans claiming the trigger "does not fire on inverse-direction prose" with the landed #1796 calibrated tokens (vm-produced / produced on the vm) — commit: f69beea9a2 Why: parked candidate on #1796 (fp aa7660c7b1fd); verified the trigger regex at scripts/verify_plan.py:6874 carries both tokens and both spans were stale.
  4. Target: CLAUDE.md — what: the task.py address-concern snippet now shows the REQUIRED --by/--round flags (a session hit argparse exit 2 following the doc) — commit: deb82542a8 Why: miner F-P4 (probed via --help: both flags required); second address-concern flag miss in 2 days. Applied edit: address-concern <N> --concern-id <id> --summary '...' → address-concern <N> --concern-id <id> --by <agent> --round <k> --summary '...'.
  5. Target: .claude/skills/issue/SKILL.md — what: (a) the Step-8 upload-verification posting site now states the required Verdict: PASS token the finalize gate regex matches (#1775 had a healthy PASS refused at teardown); (b) Step 9a-quater's blob pin is now explicitly DOC_SHA=$(git rev-parse HEAD) — never hand-extended (#1738 fabricated-SHA 404) — commit: 1c4351a69c Why: miners D-P4 (probed task_workflow.py:2468 regex) and B-P3.
  6. Target: .claude/rules/gotchas.md — what: extended the ownership-probe entry with three verified traps: probes must run in a SEPARATE Bash call from any text carrying the target literal (#1738 aborted a valid relaunch); the bracket rule applies to every pkill -f (#1482 self-killed its shell, exit 144); a false-ALIVE unbracketed pgrep hid an earlyoom kill ~50 min (#1482) — commit: a0aef434b9 Why: miners B-P5 (verified in-transcript), J-P4, D-P2.

Filed for review (route 2 — 29 tasks, each dispatched to /issue --auto or queued for the watcher backstop):

  • filed for review #1847: bash -n pin test for the Step 10d TG fenced blocks (parked candidate, #1790)
  • filed for review #1848: plan_patch.py --file alias + rc-check note (3 sessions, 2 junk plan versions)
  • filed for review #1849: hub._upload fail-loud when a file dest looks like a dir (#1738's 8-shard crash; experiment-lib, non-wf-fix)
  • filed for review #1850: remote-landing watches need a producer-fence deadline + Monitor heartbeat (3.5h + 58min dark sessions)
  • filed for review #1851: step9c selector misses SKILL.md-pinning tests (merged twin candidates from #1804 + #1813)
  • filed for review #1852: env-pin allowlist PYTORCH_CUDA_ALLOC_CONF + help example (parked candidate, #1803)
  • filed for review #1853: watcher leaves filed infra tasks undispatched for hours (10 tasks probed 8–22h at proposed; urgent-router dedup branch never dispatches its target)
  • filed for review #1854: /daily route-1 code applies run full-ruleset lint (root cause of the 40h red)
  • filed for review #1855: context-headroom check at review-round boundaries (#1776 context-ceiling death)
  • filed for review #1856: kill-before-relaunch enumerates the live launcher's remaining phases (#1482 self-inflicted OOM crash)
  • filed for review #1857: root-code guard binds cert to the staged blob (transient worktree-hash false-block; recurred on tonight's own gate run)
  • filed for review #1858: RunPod failover launcher timeout-bounds sync + reaps stale index.lock (#1769 fu1 1.5h wedge)
  • filed for review #1859: repo-root guard allows timeout-prefixed ssh (the guard's own header names this as a future-demand candidate; demand arrived ×2)
  • filed for review #1860: humanize loop verifies the candidate body BEFORE set-body (#1775 briefly-live FAIL body)
  • filed for review #1861: repo-root guard recognizes cd-to-worktree via variable indirection (2 false blocks, 2 sessions)
  • filed for review #1862: detached-VM probe template mandates bracketed pgrep; choom=failed retry for big-RSS phases (#1482)
  • filed for review #1863: poll ticks compute rate/ETA on long same-phase tails (#1482's 5.4h idle-GPU upload; source fix pre-filed as #1824)
  • filed for review #1864: analyzer.md numbers the payload-lint gate as a pre-commit step (#1775 lint-red analyzer commits)
  • filed for review #1865: pod relaunch shape fully detaches stdio (#1768's 2.5h hung ssh wrapper + 2.7h marker gap)
  • filed for review #1866: SLURM completed-work-but-FAILED disambiguation (fellows job 15194; residual beyond #1835/#1836)
  • filed for review #1867: task.py address-concern accepts --note alias (second same-flag miss in 2 days)
  • filed for review #1868: Step 10d teardown re-runs sync on the state=in-flight advisory (#1792 ended with its push unconfirmed)
  • filed for review #1869: deferred-commit sidecar captures the Failed hook line (33 undiagnosable rows in ~36h)
  • filed for review #1870: sync KEPT-stash outcome gets durable surfacing (stdout-only today; triage itself tracked in #1736)
  • filed for review #1871: judge-monitor reads are digest-grain (a #1776 tick turn was refusal-killed after cat-ing raw judge rationales)
  • filed for review #1872: register step 5a in workflow.yaml steps enum (SKILL.md names it 24×; poster errors + coarsened records)
  • filed for review #1874: /daily filed bodies cite a marker per mechanism claim or carry the unverified label (two #1677-class critic-round burns from 07-28 bodies)
  • filed for review #1875: /issue Step 0 preloads TaskOutput/Monitor schemas (3 sessions paid the deferred-schema trap in one day, post-reminder)
  • filed for review #1876: compliant git-commit forms at SKILL snippet sites + the piped-git guard's block message (≥9 blocked compositions today)

Other problems & notes

  • Route 3 (held, needs you): #1873 — pod-1768 (8×H100, ~$25/hr class) live-probed at 0–11% GPU util / load 13.7 during its ~17h-projected CPU-bound p8 fit phase (carve-out: spends money/compute). Suggested action in the task; a steering epm:progress note was posted on #1768 for the owning session.
  • #1823 is moot — its one-line fix landed inline tonight (7202cfd669, marker posted); recommend archiving it (its duplicate #1843 was already archived in-fleet). The 22h undispatched gap it exposed is #1853.
  • Parked candidate on #1771 deliberately NOT routed (mechanize the architectural park in task.py) — deferred per workflow-fix-on-bug § "target under active change" while #1771 itself (still plan_pending) rewrites that gate; it will re-enumerate, which is correct.
  • Parked candidate suppressed by eyeball: refusal-ladder prior-day-history clause — #1797 (completed today) landed a refusal-ladder revision for the same #1769 incident family; the residual clause (consume prior kill markers before rung (b)) is noted here rather than double-filed on the same surface in one day.
  • Step C sweep hygiene: 9 parked candidates enumerated, all dispositioned (5 filed via #1847/#1851/#1852, 2 self-applied, 2 resolved-upstream); 1 benign skipped row (json-decode on #1333's events, relevant_kind: false — the known shape); unmatched_record_fps advisory at ~300 refs (advisory-only, the #1703 standing condition; grows with each day's routed-records).
  • Retraction re-check: 9 marker-mined parks scanned, 2 skipped — #1791's dotenv fix already landed (load_dotenv() present, test passes) and #1795's E501 premise resolved by tonight's fix; both recorded as routed-records on their source tasks.
  • Driver suspect-probe calibration: 15 landed-fix/closed-sibling suspects flagged across the batches; 14 were eyeballed token-overlap false positives (retried per the #1735 calibration note — hot workflow files make token-overlap suspects mostly noise), 1 accepted (above). All eyeballs recorded before --retry-suspects.
  • Step B: 0 un-reconciled living-docs rejections (the single grep hit on #713 was a quoted string inside a code-review note; exact-kind check negative); 0 parked-redundant on_hold follow-ups.
  • title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-25.md). Read-only sweep exited 0.
  • Recurrence tallies (guards/protocols worked; log-only): piped-git compositions blocked ≥9× across ≥6 sessions (mechanical dent filed as #1876); foreground sleep-poll shape ×2; TaskOutput/SendMessage deferred-schema or param slips ×4 (filed #1875); Edit/Write-before-Read ×3; cwd-relative path slips ×2 sessions; API-529 storms ended turns/sessions ≥5× (tick/watcher recovered every one, incl. #1775's death with uncommitted fixes that a successor re-landed); 2 refusal kills recovered by the documented ladder.
  • Self-recovered incidents already routed in-session (verified, no re-file): #1834 (plan_deviations schema, completed), #1824 (upload per-file walk), #1835/#1836 (fellows lane), #1837 (GCP wedge 2-tick alarm), #1840/#1841 (#1768 poller/tick), #1842 (Batch custom_id charset), #1845 (batched-solve gotcha), #1846 (orphaned-JSON WARN) — all but #1834/#1838 still undispatched at proposed, which is exactly #1853's evidence set.
  • Notable session recoveries: #1768's 8×H100 sat ~16h behind a wedged HF download (found via an API 500, not design — #1840/#1841 own the fix); #1482's recovery collided with its own launcher (filed #1856) after which the E3 twin resume-skip surfaced only as an upload WARN; #1738's B1 upload-prefix crash (filed #1849) cascaded into a 3.5h deadline-less watch stall (filed #1850).
  • A-P3 (analyzer trims to exactly the cap, 3 verify-FAIL rounds) and A-P6 (judge truncation residue AT the rule-23 300-token floor, recovered at 600) — noted for the nightly consolidate_lessons cron / as measured points; rule 23 already carries the floor-is-a-floor caveat.
  • Miner coverage: 59 transcripts (10 parallel miners, groups A–J), 103 problems catalogued in logs/daily/mining-2026-07-29/; zero human-correction problems (all sessions in-window were autonomous).
  • daily stub + enrichment pushes confirmed landed on origin/main.

Living-docs drift

  • living_docs.py check: FAIL — 199 findings (relates_to ⇄ evidence mismatches, completed clean-results absent from question evidence, stale spec-role-header State trailer) — count identical to the 07-24..07-28 runs; hash shifted 078285c14444 → 1c43568fcc3d (a few evidence rows changed) but materially the same drift.
  • drift proposal already open (logged 06-29/07-04/07-06/07-07); not re-proposing. Event-stream row appended with the new hash.

My thoughts

Highlighted results

  • no results promoted today