EPS
← All docs·Activity

Daily — 2026-07-24 (26 infra fixes merged;

updated: 2026-09-03

What happened

  • The 07-23 wf-fix wave landed end-to-end: ~26 kind: infra tasks (#1640–#1666) ran the full autonomous pipeline and ALL completed + merged same-day — among them the daily-pin-test registration (#1645), the bare-commit pathspec lint (#1648), gate-timeout re-sizing (#1646), the watcher completed-unmerged respawn arm (#1653), the HF LFS write-gate preflight probe (#1654), detached-run harvest breadcrumbs (#1656), and ddp_timeout defaults (#1660). #1652 correctly self-archived as a duplicate of the already-landed #1600.
  • #1586 same-issue follow-up round completed the hard way: 5 pod crashes (3× ENOSPC, a fan-out staging race, a resume-blind headroom deadlock) across relaunches 3–8, a watcher wedge-failover that replaced the pod mid-round (fresh volume, from-p0), then run-complete → upload-verify PASS r2 → analyzer fold-in → PR #1430 merged → re-parked awaiting_promotion. GPU-h ~58 actual vs 28 budgeted (recorded honestly). wf-fixes #1641/#1642/#1664 filed from inside the round and all three merged same-day.
  • Inline user-chat rounds: #1092 pooled-probe-transfer round closed out (committed + figures regenerated); #825 got a user-requested audit of the Track-M "two-turn collapse" (verdict: ~70% fitting-machinery/n-sensitivity, not data — 010d4b7bf1); a deep-dive session unpacked what the general-linear coordinate-change claim licenses (#825/#1345/#1639, read-only).
  • Fleet-wide: Codex quota sentinel live until 2026-08-06 — every review site all day ran single-Claude (tracked in blocked #1472); zero promotions today; #1622's Step 10d merge is still stranded (PR #1399 open draft — see held #1685).

Applied workflow improvements

  1. Target: .claude/rules/gotchas.md — what: two new gotcha entries (grouped: both doc-only monitoring/probe traps from today's sweep) — commit: 9a6d4cbbc1 Why: (a) #1645's Step 10d improvised git merge-tree --write-tree probe always fails on this VM (git 2.34.1 < 2.38; rc=129, 1 firing, session 5349b9f5); (b) #1586's SSH tick read TRACEBACKS=63 on a healthy run — all non-fatal httpx AsyncClient.aclose teardown noise — risking a false crash diagnosis (session dd0af0ae, 08:38Z). Applied edit: 
    + - **VM git is 2.34.1 — `git merge-tree --write-tree` (needs git ≥2.38) is unavailable...** RULE: probe mergeability with a scratch `git merge --no-commit` in a detached worktree, or the legacy three-arg form. (Incident #1645 Step 10d, 2026-07-24.)
+ - **Non-fatal `ERROR asyncio: Task exception was never retrieved` rows (httpx `AsyncClient.aclose` teardown) inflate raw traceback counts in pod workload logs...** RULE: switch the monitoring tick to a fatal-signature grep before diagnosing a crash. (Incident #1586 fu round, 2026-07-24: TRACEBACKS=63 on a healthy run.)
    Verification: repo-wide workflow_lint.py --check-references PASS after commit.
  2. filed for review #1667: watcher wedge-failover must check for a live owning session mid-crash-fix-round before terminating a pod, and post the failover on the issue's events (autonomous_session_watch.py; #1586 05:33Z incident — owner misdiagnosed the loss for ~90 min).
  3. filed for review #1668: adopted failover pod must clear/bypass the runpod-wedge already-handled sentinel (backend_poll.py:1917) — the poll/tick lane stayed wedge-blinded all run; orchestrator hand-rolled direct-SSH ticks.
  4. filed for review #1669: watcher failover re-provision must carry the original launch env pins (WANDB_PROJECT loss → 3 runs synced to the wrong WandB project, upload-verify FAIL r1).
  5. filed for review #1670: session-reconcile pass must never stop a session younger than the idle threshold — it killed the watcher's OWN 10-min-old #1622 merge-recovery respawn (idleness read the task's 27.5h marker gap, not session age).
  6. filed for review #1671: sync_repo_root.py must bounded-poll + retry rebase --abort on the HEAD.lock race instead of exiting 6 with the shared root detached mid-rebase (#1645 session; 40+ unpushed commits until manual recovery).
  7. filed for review #1672: Step 10d merge-recovery in a stale worktree must sync lint cross-reference targets before committing (#1622: plan_patch.py missing from a 1240-behind worktree failed pre-commit lint; ~25 files ad-hoc synced).
  8. filed for review #1673: inline-round worker briefs staging repo-root .py payloads must carry the inline_lint_gate.py cert command + report guard-blocked commits immediately (#1092 pooled-probe-runner stalled 3× on the last-mile commit).
  9. filed for review #1674: daily_drive_filings.py runs a mechanical landed-fix git-log probe at filing time (3rd recurrence of filed-over-landed: #1330/#1386/#1652-over-#1600).
  10. filed for review #1675: guard_piped_git_push.sh false-blocks read-only greps whose QUOTED pattern mentions git commit — anchor the match to pipeline-segment command position (07-23 /daily run evaded the hook by rewriting its pattern).
  11. filed for review #1676: guard_root_code_commit.sh blocked a cd $WT && pytest && git commit compound as a repo-root commit — recognize the cd-into-worktree prefix or name it in the BLOCKED message (#1644).
  12. filed for review #1677: /daily route-2 bodies must label non-grep-verifiable numeric/root-cause claims as unverified hypothesis — 4 of 6 wave bodies carried refuted premises, each burning a correction round (#1655/#1662/#1646/#1660).
  13. filed for review #1678: driver detects same-target-file manifest siblings and staggers/WARNs — 2 Step 10d merge conflicts today were same-wave same-file contention (#1645×#1659, #1655×#1643).
  14. filed for review #1679: extend verify_plan.py c37 — plans adding a workflow_lint check must name a *bundled_in_no_flags* pin test (routed from #1648's parked candidate, Step C).
  15. filed for review #1680: Step C suppression must also honor origin_candidate_ts for fp-bearing candidates (last night's routed-records carried driver-recomputed fps → all three #1630 candidates re-enumerated tonight; corrective re-keyed markers posted) + surface skipped_rows with reasons (the #1642 park was silently absent from tonight's sweep).
  16. filed for review #1681: parked wf-fix candidates flagging a LIVE red test on main get an urgent — main is red routing path instead of waiting for the nightly cycle (#1643's park: ~11.5h of fleet-wide gate noise).
  17. filed for review #1682: implementer report contract — SHAs pasted verbatim from git rev-parse, never hand-extended (#1586 round-7 report shipped a fabricated hex extension).
  18. filed for review #1683: reconcile tests/test_issue811_pre_user.py with the current issue667_extract.py API — 5/21 red on pristine main (experiment-code drift; daily-auto-filed only, no wf-fix tags).
  19. filed for review #1687: route-3 overlap dedup matched on generic workflow tokens (gate/step/warn) and wrongly suppressed tonight's issue823 held filing against unrelated #1537 — extend the informative-token stoplist (task_workflow.py).

Other problems & notes

  • Held (route 3, needs-human) #1684 — matched-n control for cross-cell R² comparisons. Carve-out: scientific-meaning change. The #825 "two-turn collapse" was presented as a data property; the user-requested audit showed ~70% is fitting machinery/n-sensitivity (0.482@n=700 → 0.258@n=2000 → 0.673@n=5000 on matched-n Track S). Proposal: standing rule requiring a matched-n subsample control before a lower cross-cell R² is narrated as a corpus/condition property. Suggested action: approve wording → one /issue session wires planner/critic surfaces.
  • Held (route 3, needs-human) #1685 — re-drive the stranded #1622 merge (PR #1399, open draft since 07-23). The watcher killed its own recovery respawn (see #1670) and the once-per-episode respawn budget is consumed. Options: respawn now (spawn_session.py spawn-issue --issue 1622 --auto, risks a repeat kill until #1670 lands) or wait for #1670.
  • Held (route 3, needs-human) #1686 — disposition the untracked root draft scripts/issue823_single_split_protocol.py. Carve-out: destructive. ≥11h on the shared root; 4 Step 9c gates today degraded to scratch-oracle mode over it. Ownership check first, then commit-to-owner or delete. (Driver initially suppressed this as "already tracked in #1537 (shared: gate,step,warn)" — a wrong match on generic tokens; filed directly via the wrong-match escape; the dedup weakness itself is filed as #1687.)
  • Step C parked-candidate dispositions (6 sweep rows + 1 sweep-missed): #1630's three candidates → deduped against LANDED #1645/#1648 (re-keyed routed-records posted with sweep-canonical fps); #1643's park → skipped filing: premise retracted upstream (the flagged test now PASSES on main — the #1586 dotenv fixes landed via PR #1430 + #1666); #1648's park → filed #1679; #1654's park → deduped against #1654's own landed r2 fixture (0901608742); #1642's park (absent from sweep output, found manually) → deduped against LANDED #1646 (gate-timeout re-sizing, merged same day).
  • Retraction re-check: 7 marker-mined problems scanned, 1 skipped (retracted), 4 deduped against landed fixes. Transcript-mined filings were current-tree confirmed (e.g. the issue823 draft still present; PR #1399 still open; issue811 symbols still absent).
  • Codex quota outage (tracked in blocked #1472): the ENTIRE day's ~26-task infra wave + all #1586 review rounds shipped single-Claude (zero cross-family diversity); sentinel until 2026-08-06. Freshness note posted on #1472, including a session's suggestion to delete the sentinel once mid-window to probe for an early quota reset.
  • Two one-turn Usage Policy refusal kills, both self-recovered (dd0af0ae 13:14Z post-critic-PASS; f0486147 11:58Z pre-review — tick backstop re-drove in ~40s). Below the wedge threshold; no action.
  • #1586 GPU-h overrun recorded honestly: ~58 actual vs 28 budgeted (~2× — five crashes + a from-p0 failover relaunch). Context for promotion review.
  • Hook/guard true positives worked as designed (piped gh pr merge blocked → bare re-run; repo-root rebase blocked → -C $WT form; heredoc commit slip → one retry). No action beyond the two FP filings above (#1675/#1676).
  • Opus-5 recency miss (session 1e919923, no task): the assistant confidently denied the same-day Opus 5 release from stale bundled docs; corrected after user pushback + web search. Global-behavior fix (web-check unknown models before denying existence) belongs to the user-global surface — flagged for /memory-sleep, out of this skill's project-file scope.
  • staged-set audit: clean on tonight's commits (only this run's own files staged each time).
  • title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-11.md for verbatim rows).
  • Driver SHA-scanner noise: several WARNING <slug>: hex token ... does not resolve as a commit lines were session-id citations (dd0af0ae, 8ca48206, ...) correctly labeled as sessions in the bodies — no action.

Living-docs drift

  • living_docs.py check: FAIL — 199 findings (≈75 dead legacy anchors; ≈54 completed clean-results in no question's evidence; relates_to ⇄ evidence mismatches incl. #722/#763/#494–#501; spec-role-header State trailer stale vs 7 promoted results).
  • Drift hash 078285c14444 — byte-identical to the 07-07/07-22/07-23 rows.
  • drift proposal already open (logged 2026-06-29/07-04/07-06/07-07); not re-proposing
  • Step B: #713's declined living-docs proposal remains the only un-reconciled rejection; no context change today bearing on it — skipped silently (no parked-redundant on_hold follow-ups exist).

My thoughts

Highlighted results

  • no results promoted today