EPS
← All docs·Activity

Daily — 2026-07-04 (10 experiments to awaiting_promotion, 66 infra fixes landed, 0 promoted)

updated: 2026-07-05

What happened

  • Huge autonomous throughput day: 4,143 commits across 113 tasks. 10 experiment lines reached awaiting_promotion (#742, #810, #825, #841, #922, #923, #928, #931, #952, #958) and ~66 tasks reached completed — almost all of them the workflow-fix burst filed by the 07-01→07-03 daily backfills (#960–#1025 range), each through the full plan/critic/code-review pipeline.
  • The 07-02 and 07-03 daily backfills both completed today (logs/daily/2026-07-02.md, 2026-07-03.md), the second after recovering 10 filing bodies stranded in /tmp by the killed first attempt (#994's failure class).
  • Two fleet-level incidents, both closed same-day: (a) a stranded rebase husk detached the shared repo-root HEAD twice (~09:20 and ~21:30 UTC), crashing task.py reads fleet-wide — fixed by #971 (husk recovery) + #999 (graceful degradation); (b) earlyoom memory-pressure kills at 06:38 UTC and through the evening SIGTERM'd test gates and long fits in at least 5 sessions (filed as #1045 tonight).
  • #931 committed a workflow-lint offender to main that broke test_workflow_lint on pristine trunk for most of the day — ≥5 unrelated sessions burned 5–25 min each classifying it pre-existing before it was fixed (pre-push payload lint filed as #1047).
  • Experiments in flight at day end: #811/#813/#923/#952 in same-issue follow-up rounds (#811's serial ridge fit ran ~19h at ~4× its plan estimate before the session pivoted to a vectorized relaunch — the recurring vectorize-first failure), #833 interpreting, 4 daily-fix tasks in planning/review.
  • No clean-results were promoted today; the awaiting_promotion queue stands at 83.

Applied workflow improvements

  1. Target: .claude/skills/adversarial-planner/SKILL.md — what: added a "Read a Bash-materialized plan copy before Editing it" line to the plan-logging bullets (restates the harness invariant at the spot where revision rounds create plan copies) — commit: 0cf63216ed (rewritten from local 5a96f74005 by a concurrent pull-rebase during the push; 0cf63216ed is the sha on main — revert with either, both objects carry the same diff) Why: the #1006 session fired 7 consecutive Edits at a just-copied plan file, all bounced with "File has not been read yet" (10:54–10:55 UTC, session 69b2e3fb); 11 occurrences of the same error class across 5 sessions today. Applied edit:

    + - **Read a Bash-materialized plan copy before Editing it.** When a revision round creates the draft via Bash (`cp plans/plan.md /tmp/...`, a heredoc, or a python writer), the harness requires an in-session `Read` of that file before any `Edit` — firing Edits straight at a just-copied file bounces every one with "File has not been read yet" (7 consecutive bounces in one 2026-07-04 session). Read once, then edit.

    Verification: workflow_lint.py --check-references PASS, --check-asks PASS (repo-wide, post-commit).

  2. filed for review #1044: sync_repo_root husk liveness probe + "Cannot rebase onto multiple branches" internal retry (residual seams after #971/#996/#1030).

  3. filed for review #1045: choom-protect Step 9c gates + VM long-fit launches by default (earlyoom killed #906/#995/#811/#742/#813 work today).

  4. filed for review #1046: Step 9c gate timeout sizing by file count + slow-file batching + recipe exit-code hygiene (incl. the 9a-quater gist snippet's false Exit 1).

  5. filed for review #1047: Step 10d hardening — pre-merge commit of dirty agent-memory files, pre-push workflow-lint on the merge payload (the #931 trunk-breakage class), sync_repo_root.py as the only root-sync recipe.

  6. filed for review #1048: mechanical guard blocking piped git push (the masked-rejected-push class recurred in #957).

  7. filed for review #1049: document the pod→HF upload wedge escalation ladder (xet → hf_transfer → VM rsync reroute, from #931's ~50-min recovery).

  8. filed for review #1050: orchestrator runs the mechanical verifiers and inlines output for the Codex clean-result critic (its read-only sandbox can't run uv — #841 round 4 shipped without the pre-pass).

  9. filed for review #1051: tick_triage liveness probe before STALE-REDRIVE (9 false re-drives in 3.5h against #931's healthy multi-hour fit).

  10. filed for review #1052: CRON-TEARDOWN sweeps one-shot wakeups + idempotent CronDelete (#980's completed task nearly got re-driven; #988 hit a stale cron id).

  11. filed for review #1053: Step 0 detect-and-exit posts an audit marker before ending (the silent 23-line #958 session corpse).

  12. filed for review #1054: new_worktree.sh pre-adds the issue's own eval_results/issue_<N>/ sparse cone (#906's artifact commit bounced).

  13. filed for review #1055: GCP finalize false-failed phase when deliverables are verified complete (two #811 diagnosis cycles today; complement of #1004's false-done).

  14. filed for review #1056: clean-result critics verify figures against the SHA-pinned blob, never untracked repo-root strays (#922 burned a reconciler round on it).

  15. filed for review #1057: guard_log_dump exempts code files / gates on requested-range size (blocked a legitimate range read of workflow_lint.py in #986).

  16. filed for review #1058: repo-root guard false-positive on worktree-scoped checkout recipes (3 sessions today + this run's own generator heredoc got blocked).

  17. filed for review #1059: stagger proposed-infra-sweep session dispatches ≥60s (5 spawns in 3 min tripped the org 4M input-TPM cap repeatedly).

  18. filed for review #1060: plan-compute-sizing requires a measured 1-cell pilot timing for per-cell fit phases + a 2×-deviation vectorize trigger (#811 ran ~19h vs a ~6h estimate).

  19. filed for review #1061: /daily writes filing bodies to a durable dir + files incrementally (the 07-03 attempt stranded 10 bodies in /tmp; tonight's run already adopted the durable dir ad hoc — codify it).

  20. filed for review #1062: de-escape HTML entities in plan-handoff workload commands (#952's handoff carried an escaped shell operator; low confidence on root cause).

  21. filed for review #1063: apply the prefix+context both-arms mapping default at task-CAPTURE time (Thomas had to correct #958's capture 6 minutes after filing it).

  22. filed for review #1064: artifact-reuse fitness check verifies the consumer's read path against the staged layout (#928's Store() crash on the flat Hub prefix).

Other problems & notes

  • Route-3 held → #1065 (needs-human): Step 10d task-state merge-conflict contract. Every long-lived issue branch now conflicts at merge because tasks/<status>/<N>/ paths git-mv on main (#953, #967, #906 today — #906 was 6,738 commits behind). Decide: exclude tasks/** churn from issue branches (public-contract-adjacent) or bless recovery-merge as the normal path. Carve-out: contract change / ambiguous intent.
  • Route-3 held → #1066 (needs-human): #779-line terminology + writeup conventions. You re-steered the #779 writeup across ~19 messages (context-vector/query-vector naming, PCA=64 rationale, KRR-as-nonlinearity-probe). Confirm the settled terms so they land in docs and future writeups pre-justify method choices. Carve-out: taste / scientific meaning.
  • Route-3 held → #1067 (needs-human): Happy mobile push was down ALL DAY — every notification (26 in one session) fell back to terminal-only with "Remote Control inactive". Only you can re-pair the phone channel. Nothing urgent fired today, but delivery would have silently failed if it had.
  • Registry/filesystem drift GREW — beyond the #696/#722/#750/#764/#766 set already held in #801, today's audits also name #698/#699/#702/#708/#709 (incl. #709 at planning, invisible to REGISTRY consumers). Posted an addendum marker on #801 scoping the human-audited repair to the full current audit; not self-repaired (respecting the prior route-3 hold).
  • Spurious usage-policy refusals killed 6+ subagents across #958 (×2), #931 (×2), #952, #965 — every one recovered via the documented durable-writes-first + rephrased-retry ladder. No new fix derivable.
  • Foreground sleep-poll guard blocks: ~10 occurrences across 8 sessions (down from 15+ on 07-03); the guard held every time. Deferred-tool schema misfires (Monitor/TaskOutput called without ToolSearch): ~7 occurrences, all self-recovered — the CLAUDE.md rule exists; model-compliance, not a workflow gap.
  • The Claude+Codex ensemble caught 3 real implementation bugs Claude-only review missed today (#1019's out-of-band-canceling adoption, #928's silent-stale-resume hole, #997's fix containing the very defect it fixed) — plus ~20 reconciler rounds; machinery working as designed.
  • #982 ran as a duplicate (its exact fix had landed via #974) — wording-sensitive fingerprint dedup miss; the clarifier's "grep main first" caught it at Step 1, cheap. No clean fix derivable.
  • The "identical"-with-fingerprint humanize false-positive (#958) is a sibling of the verbatim-sample exemption already filed as #998 — noted there rather than filed separately.
  • GCP manual-clone secrets gap (#833: 3 invisible rc=78 boot poweroffs from a missing WANDB_API_KEY mirror) — adjacent to #1029 (GCP pre-workload boot failures), which is in planning; its planner should see this incident.
  • The #906 production-400-behind-faked-seam lesson (live wire probe before first --full) is covered by the already-filed #985 + #1007; not re-filed.
  • Fleet substrate notes: root-disk pressure fully drained today (56 GB free, healthiest in 2 days); Anthropic 429/529 storms (~40 events, 8 sessions) all absorbed by auto-retry; HF org 429s hit both ad-hoc unscoped listings (#811, #923) — code-side scoping is #988's scope.

Living-docs drift

  • living_docs.py check FAILs with 203 findings: 4 registry/filesystem inconsistencies (#696/#750/#764/#766 — held in #801, see notes above), ~75 relates_to pointers to dead legacy anchors (a1–h3 era), ~50 evidence-list asymmetries, 54 completed clean-results in no question's evidence, and a stale spec-role-header State trailer vs 7 newer results.
  • drift proposal already open (logged 2026-07-04 by the 07-02 backfill, hash d2a80db46b3b); not re-proposing.
  • Parked-proposal re-consideration: no unreconciled living-docs rejections (the one grep hit on #713 was prose, not a marker) and no parked-redundant on_hold follow-ups eligible for revival.

My thoughts

Highlighted results

  • no results promoted today