EPS
← All docs·Activity

Daily — 2026-07-16 (1 result promoted; 5 fixes applied, 34 filed)

updated: 2026-09-03

What happened

  • #1073 promoted useful (04:25Z): a single greedy answer is an adequate stand-in for 10-rollout-averaged answers — trims the #952/#1092 line's eval cost.
  • A ~35-task workflow-fix wave completed and merged (#1363–#1433 range): MALLOC_ARENA_MAX=2 VM-launch env (#1363), EXDEV hub-staging gotcha (#1427), hf-xet wedge gotcha (#1425), 10 torch-before-dotenv figure scripts fixed (#1428), hub-verify waivers (#1388), stale-status husk dirs filed (#1430), pilot-gate designed-halt gotchas (#1431), sweep-8g-a100 width degrade (#1379), repo-root guard ssh mask (#1413), stale-pod ownership gate (#1404), and more.
  • Experiments advanced to awaiting_promotion: #1310 (recovered a p5_upload crash via crash-persist partials), #1315, #1333 (two launch-composition crash-fix rounds), #1336 (E1 dispatch lane fight + G1-ordering fix). #825 round 10/11: token-budget-death root cause fixed with the realistic-brevity simulator v2; round-11 launch refused on a foreign-tenant GPU holding 72 GB (candidate routed). #1005 survived a GPU-memory-leak OOM (fix round 4) and a cross-issue upload-prefix clobber of #928's artifacts (re-homed + restored). #1090's follow-up scope was broadened into two parallel tracks; #1415 burned a GCE cycle on a batch-shape-mismatched pilot gate (fixed + gotcha'd).
  • Fleet conditions: Codex org quota exhausted ALL DAY (sentinel until ~Aug 6) — every doubled review site ran single-Claude; org-wide 429 output-TPM storms (~16:05–16:12Z) burned 27–112 turns per session across ≥7 sessions; / hit 98% twice (home HF cache ballooned to 101 GB); the tmux server socket was deleted from /tmp, silently forking a second server (39 sessions invisible until recovered).

Applied workflow improvements

  1. Target: .claude/rules/analyzer-section-reference.md — what: synced both byte-identical ban elaborations to the full bit/-equal family (#642 drift) — commit: e6f1472d8f Why: parked candidate on #1423 (code-reviewer r1): lines ~489/~792 named only byte identical/byte-identical while the audit regex bans the (byte|bit)[ -](identical|equal) family. Applied edit: 
    - **Never write `byte identical` or `byte-identical`** anywhere in the body (banned 2026-W22, task #454; carried into v3; ...)
+ **Never write `byte identical` / `byte-identical` — nor the `bit` or `-equal` family variants (`bit identical`, `bit-identical`, `byte-equal`, `bit equal`, ...)** anywhere in the body (banned 2026-W22, task #454; family widened by #642; ...)
  2. Target: .claude/rules/pod-side-reporting.md — what: cross-referenced the gotchas designed-halt pilot-gate convention (gate refusal = report JSON + distinct rc, never bare rc=1) — commit: 0369301bb5 Why: parked candidate on #1431 (Alternatives critic): the dispatcher-side sibling surface never pointed at the new gotchas entry. Applied edit: 
    + **Designed-halt exit codes:** a plan-registered gate refusal (a pilot timing gate, any stop criterion) is NOT a crash — the dispatcher writes a gate-report JSON and exits a DISTINCT rc ... Full convention + the #1415 incident: `.claude/rules/gotchas.md` § pilot timing gates.
  3. Target: scripts/select_step9c_tests.py — what: fixed the stale recommended_timeout_s docstring figure — commit: b8a6300b4e Why: parked candidate on #1397: docstring said 2130 s; the live invariant-only figure is 2160 s (verified by executing recommended_timeout_s(sorted(WORKFLOW_INVARIANT)) → 2160). Applied edit: 
    -    workflow-lint surcharge) -> 2130 s (~35.5 min), consistent with the existing
+    workflow-lint surcharge) -> 2160 s (36 min), consistent with the existing
  4. Target: .claude/skills/paper-plots/SKILL.md — what: transcribed the standing no-plot-annotations rule into the skill (new § 3.4-bis) — commit: 5538de711a Why: Thomas had to re-correct it on the #1345 dashboard ("remove the text on top of the ... bar", 09f28ede ~08:11Z); the rule lived only in auto-memory, which doesn't load in every plotting context. Applied edit: 
    + ### 3.4-bis. No explanatory overlays on plots
+ Keep figures clean: **no arrows pointing at features, no effect-size labels stamped on/above bars or points, no explanatory text overlays** ...
  5. Target: CLAUDE.md — what: named background-Bash sleep-loop watchers as the same banned polling shape as foreground sleep chains — commit: c63bd72fc4 Why: a #1310-followup session (f2378bdb, ~08:57Z) launched two bg sleep-loop watchers the harness killed before giving up; the bullet named only the foreground form. Applied edit: 
    + A **background**-Bash `sleep`-loop watcher (`for i in $(seq 1 30); do sleep 300; ...; done` under `run_in_background=true`) is the SAME banned shape — the harness kills those too (2 killed watchers in one #1310 session, 2026-07-16); the sanctioned pollers are `poll_pipeline.py` bg-Bash and `Monitor` until-loops.

Filed for review (route 2 — each lands via its own /issue --auto pipeline; most spawns stagger-deferred to the watcher backstop). From the Step C parked-candidate sweep (21 parked candidates routed):

  1. filed for review #1437: allowlist issue952_behavior_dashboard.py in the pod-shellout invariant test — the ONE red test on pristine main, raised independently 4× today (#1392/#1402/#1407/#1433 + prose on #1397/#1399/#1424).
  2. filed for review #1438: codex-code-reviewer rubric parity (Step 0.9 copy-list + data-access-blocked tag enumeration) — 2 parked candidates on #1380.
  3. filed for review #1439: periodic task.py audit sweep for REGISTRY drift (watcher) — #1382.
  4. filed for review #1440: task.py poster-side WARN on v<k>. -stamped note heads — #1382.
  5. filed for review #1441: workflow.yaml L3255 BINDING summary gains the #1383 context-consistency clause — #1383.
  6. filed for review #1442: TG_TESTS pytest blocks routed through the step9c tmproot/--basetemp — #1408.
  7. filed for review #1443: anchor the repo-root guard's cd-scope latch to clause-initial cd — formal candidate on #1413.
  8. filed for review #1444: Methodology lens gains the out-root mount-binding item #1414 landed rule-side — #1414.
  9. filed for review #1445: SKILL_REF_RE filesystem-path (/tmp) false-positive carve-out — #1414.
  10. filed for review #1446: widen the recently-closed-sibling advisory to non-prefixed kind:infra tasks (the #1386-over-#1360 mechanical half) — formal candidate on #1420.
  11. filed for review #1447: widen bit_byte_identical to the synonym tail (byte-exact, bit-for-bit, bitwise identical) — #1423.

From transcript mining (8 parallel miners over 127 transcripts):

  1. filed for review #1448: 429-retry Stop hook — limiter-aware message + minute-boundary pacing + re-wake cap (7 sessions burned 27–112 turns each today; 99–150 instant hook re-wakes in the worst window). wf_fix: false (global hook).
  2. filed for review #1449: plan-glob vs uploader allow-pattern parity check (#825's 404 row-index JSONLs were never upload-eligible).
  3. filed for review #1450: size-cap trigger for the home HF cache reap (age-only gate let 101 GB accumulate; / at 98% twice today).
  4. filed for review #1451: lint .claude/rules/*.md frontmatter parses + paths: shape (the "rule present but never loads" class, #1385).
  5. filed for review #1452: lint hardcoded cross-issue upload prefixes (#1005's tensors overwrote parent #928's HF artifacts).
  6. filed for review #1453: watcher instant wedge lane for Prompt is too long (#1335 lost ~65 min wedged).
  7. filed for review #1454: --tunnel-through-iap in GcpBackend.fetch_results (crash-forensics fetch silently useless on IAP-only VMs).
  8. filed for review #1455: spawn_session.py stop atomically unregisters + releases the dispatch lease (deliberate stop→respawn took 3 attempts + manual clears tonight, #1090).
  9. filed for review #1456: pre-push lint gate reads ratchet constants from fetched origin/main (worktree-drift false failures, #1366 + #1411 same day).
  10. filed for review #1457: vm_disk_guard WARNs on device-vs-filesystem size mismatch (1 TB disk ran with a 485 G filesystem until Thomas noticed).
  11. filed for review #1458: extend the ad-hoc provenance rule — display-substitution disclosure (the ARIA→Assistant dashboard, #1345) + matched-target note on cross-arm tables ("these results seem wrong", #1092).
  12. filed for review #1459: address-concern 200-char cap UX (2 sessions burned round-trips today).
  13. filed for review #1460: lint step in the inline free-analysis same-turn completion contract (2 inline-landed lint-red scripts taxed 4 sessions' gates).
  14. filed for review #1461: neutralize planner-REVISION briefs on guard/security tasks + add steering/causal-intervention vocab to refusal rung (e) (#1413 truncated-brief dispatch; #1415 3 refusal-killed spawns).
  15. filed for review #1462: five small prevention notes (pgrep self-match bracketing; inline-snippet signature probe; launcher arg-parse probe; relaunch log rotation; per-leg regime-keyed out-roots).
  16. filed for review #1463: repo-root guard waiver for gcloud compute ssh --command payloads (2 sessions split commands silently; tonight's own /daily heredoc was also text-match blocked).
  17. filed for review #1464: cpu-* intents in _DEFAULT_GPUS_FOR_INTENT (free SLURM lanes degrade to unranked on CPU dispatches, #1336).
  18. filed for review #1465: relay pod_lifecycle stderr through runpod.py's CalledProcessError (opaque provision failures, #1336).
  19. filed for review #1466: move tmux sockets off /tmp + identify the socket deleter (39 invisible sessions today). wf_fix: false.
  20. filed for review #1467: /daily filer runs git rev-parse --verify on every SHA cited in a filing body (a non-resolving SHA cost #1414's fact-checker a round).
  21. filed for review #1468: A100-40 rung eligibility vs realistic FREE device memory (vLLM engine-init crash on 17 GB-consumed spot instance, #1315).
  22. filed for review #1469: Step 6a.5 gate asserts git-tree reachability on git-clone-only lanes (#1434 lost 24 run-attempts to an uncommitted manifest).
  23. filed for review #1470: #952 divtrain generation through api_dispatch + empty-completions-as-failures (138 lost gens as bare counts; 62 empties kept as successes). wf_fix: false.

Other problems & notes

  • Route 3 (needs-human, filed + PM-surfaced):
    • #1471 — extend the stale-pod ownership gate to non-managed EXITED pods? Held: reverses a deliberately-scoped, critic-accepted #1404 plan decision on a pod-terminating (destructive) path.
    • #1472 — review diversity during the Codex outage: every doubled review site runs single-Claude until ~2026-08-06 (org quota). Accept, or wire a temporary alternate cross-vendor reviewer? Held: spends money either way.
    • #1473 — sudo -n choom fails intermittently, so Step-10d lint/pytest gates frequently run earlyoom-UNPROTECTED (13 failed vs 8 ok in one session). Fix sudoers (system state change) or drop the warn?
  • retraction re-check: 17 parked-candidate source tasks scanned with post-candidate event re-reads — 0 suppression hits; 4 premise-resolved skips against the current tree instead (see next bullet).
  • Skipped filings (premise already resolved on the current tree): #1412's hub-verify waiver (landed via #1388 — HUB_VERIFY_RETRY_EXEMPT comments at both sites); the 10-11 torch-before-dotenv figure scripts (landed via #1428; invariant test passes); the compute-statement peak-RSS line (landed via #1414, SKILL.md ~L6346); PASS_UNIFIED smoke min-N (already filed as #1349 by #1315's own session).
  • User corrections not otherwise routed (verify-before-asserting / scope violations, no rule gap): "i thought we already had these results" (#1092 chat — banked JSONs checked only on push-back); "shouldn't you fit prefix→query" (design catch pre-dispatch); "Look at the rest of the result and update your response" (partial-read answer, #779/#823 ridge fitters); first-pass figure overshot the ask (#952 dashboard — simplest 2-axis form first); "can we run it in parallel and be faster?" (#1090 — offer the parallel split proactively).
  • Mentor-facing writeup defect (Thomas already in the loop): Dan read the #825/#952 writeup's Result 2 "fitted general-linear change of coordinates" phrasing as mathematically vacuous; the clarification was drafted in-session (5a0e92b9). Suggested action: revise the saved writeup's Result 2 phrasing to state the non-vacuous content (equal predictive power of the factored base map) explicitly.
  • Handled infra flakes (no action derivable): org-wide 429 output-TPM storms ~16:05–16:12Z (recovery hooks worked; hook pacing filed as #1448); ~24 API-529 blips retried clean; HF Hub 504 storms (one exhausted a 900 s retry budget on #1092, recovered manually); Step 10d merge conflicts ×5 (documented recovery worked; #1403's fresh-subagent path landed same day); a git fetch ref-lock race (#1419, retried); GCP FLEX_START 300 s queue-wait reconnect (#1333, worked).
  • Refusal kills (ladder worked, spawns burned): #1415 planner ×2 + implementer (sonnet-pin recovered); #1401 fact-checker (credential vocab); #1433 fact-checker; #1436 fact-checker ×2 (statistics-note vocab); #1332/#1333 residuals. Rung (e) vocabulary extensions filed under #1461.
  • Red-main residual: after #1437 (pod-shellout) lands and given #1388/#1428 landed today, re-run the full suite to confirm pristine main is green again — 3 sessions burned classification rounds on it today (#1372/#1380/#1387).
  • Foreign-tenant GPU on fresh pod-825 (72 GB held on GPU 4, invisible to compute-apps): candidate already routed by #825's session (its epm:workflow-fix-task-filed marker); #1416 landed the RunPod gate fix the same day.
  • 39 stale happy claude wrappers (since ~Jul 1) surfaced by the tmux recovery — the June-12 RAM-bloat pattern; triage/kill is pending Thomas's decision in that chat (not filed — user already engaged).
  • Title-sync drift sweep (Step D): title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-11.md for the verbatim rows).
  • Spawn-dispatch note: most of tonight's 34 route-2 filings deferred their session spawn (dispatch stagger / infra cap 5) — the watcher's proposed_infra_sweep dispatches them over the coming hours; see logs/daily/filings-2026-07-16/filed.jsonl for per-filing tails.

Living-docs drift

  • living_docs.py check FAILs with 199 findings — normalized drift hash 078285c14444 EXACTLY matches the open re-synthesis proposal recorded 2026-07-07 (extending 06-29/07-04/07-06): ~75 dead legacy anchors, relates_to ⇄ evidence mismatches (incl. #722/#763/#494–#501), ~54 unlinked completed clean-results, 7 stale spec-role-header State-trailer rows. docs/open_questions.md unchanged since 2026-07-07.
  • drift proposal already open (logged 2026-07-07); not re-proposing. The standing proposal remains: re-synthesize open_questions.md (living-docs updater/backfill) — needs your ok; not auto-applied.
  • Parked-proposal re-consideration (Step B): the one un-reconciled living-docs rejection (#713 — a workflow-consolidation task, unrelated to #1073's promotion) has had no context change — not re-surfaced; no followup-parked-redundant markers exist on any on_hold task — 0 revival candidates.

My thoughts

Highlighted results

  • #1073 — A single greedy answer is an adequate stand-in for 10-rollout-averaged answer