EPS
← All docs·Activity

Daily — 2026-07-14 (0 promotions; 4 fixes applied, 15 filed for review, 1 held)

updated: 2026-09-03

What happened

  • Very heavy fleet day: 590 task-state commits across ~33 live sessions. ~10 workflow-fix / daily-fix infra tasks completed end-to-end and merged to main (#1301–#1309, #1311 pod-side sentinel read-back tolerance, #1312 Lens-7 CJK-intrusion scan); #1313 (api_dispatch 529-transient, the llm-judging rule-24 sibling) and #1314 (verify_plan c6 reuse-map false-positive) filed from today's incidents and mid-pipeline overnight.
  • #779 (fitter-fair-comparison): Thomas caught the report still claiming n=3600 while the 07-06 n=10k round sat uncommitted for 8 days; the session then ran a user-directed n=50k scaling round on a fresh 8×H100 pod (two crash-fix rounds: an HF lazy-generator 404, then the known vLLM generate() hang — mitigated with enforce-eager + prefix-caching-off), a KRR λ×γ grid re-fit, extended + committed the scaling figure, and started dispatching an n=1M round (LMSYS + WildChat, 4 pods). A nested-CV run settled the map-vs-persona-vectors head-to-head at parity with a small hallucination edge (after the first head-to-head used the wrong, in-distribution artifact — Thomas caught it).
  • #825 (context→answer map): mentor writeup docs/mentor_updates/2026-07-14_context_answer_map_results.md built through ~15 revision rounds + gist dashboard. Inline map-alignment analysis flipped the headline: the base→instruct read-out change is a general LINEAR reparameterization, NOT a rotation. Naturalistic Track-S follow-up went through 3 GCP launches (zero-width-span crash; then an 84-min serial-CPU-MLP wedge that lost the turnstore; ridge-only relaunch in flight). Re-parked at awaiting_promotion. New #1310 (per-character mapping) filed + iterated through 3 designs to a prefill-based v3; v3 result FLIPS the v2 null (v2 was a power artifact); parked on_hold.
  • #1090 (multi-behavior datagen line): fu4 extended-dose-lr-sweep completed → awaiting_promotion (headline: the parent's "training context gates expression" was an optimization-budget artifact); fu5 impolite-organism finish round planned (v7) + implementing overnight. #1092: an inline free-analysis caught a live battery-leak bug in the headline fit filter (fit_grid.py:1841); the off-VM 4-box GCE refit was dispatched, tripped its RSS/wall pilot gate (71.9 GB vs 64 planned; ~2.6× wall) and self-recovered from checkpoints. #952: off-policy mapping report written; the dropped china-politics category rerun inline (55 new pairs, china-included stats committed) → awaiting_promotion. #928: prefix-mapping-arms round recovered from a 6-day stall, merged, title upgraded to HIGH confidence → awaiting_promotion.
  • Recurring correction of the day: follow-up routing vs inline — Thomas killed 4 just-spawned /issue follow-up sessions within ~2 min and rerouted to the inline path 5+ times ("run it inline so it is faster", "can you just run them inline"). Filed as #1323.
  • Codex twins were offline ALL day (org quota exhausted until 2026-08-06) — every review gate ran single-Claude via the sanctioned no-show fallback. Queue at end of day: 90 awaiting_promotion, 20 proposed, 111 on_hold, 3 followups_running, 0 promotions today.

Applied workflow improvements

  1. Target: .claude/rules/gotchas.md — what: new gotcha — HF Hub list APIs return LAZY generators; a try/except around the call catches nothing (error raises at iteration) — commit: 4474005ef4 Why: #779 n50k first launch (session 272c80a1, 23:08Z): "_remote_index DID have a try/except, but list_repo_tree is a LAZY generator" — the 404 escaped the handler and crashed all 8 shards (~25–35 min of 8×H100 lost). Applied edit:

    + - **HF Hub list APIs (`list_repo_tree` et al.) return LAZY generators — a try/except around the CALL catches nothing; the HTTP error raises at ITERATION time.** Materialize (`list(...)`) inside the try/except... (Incident #779 n50k, 2026-07-14 ... fixed at a070c8bab7.)

  2. Target: scripts/pod_lifecycle.py — what: reworded the self-contradictory host-pinned resume error ("can't be retried" + "wait and re-run") to one clear recommendation (fresh provision; retry only for irreplaceable volumes; do NOT arm a retry loop) — commit: b2e6638b54 (string-only; ast.parse + ruff clean) Why: #779 (session 272c80a1, 21:38Z): a 24-attempt resume-retry loop was launched against the constraint the tool itself called non-retryable; Thomas cut it off — "i don't htink it will resume. Just provision fresh." Applied edit:

    - ... Resume never relocates a pod, so this can't be retried. Either wait for capacity to free up and re-run `pod.py resume ...`, or provision a FRESH pod ...
+ ... Resume is host-pinned (it never relocates a pod), so retrying helps only if that exact host frees up — usually it doesn't soon; do NOT arm a retry loop. RECOMMENDED: provision a FRESH pod ... Wait and re-run `pod.py resume ...` ONLY if the stopped volume holds irreplaceable data.

  3. Target: .claude/rules/plan-compute-sizing.md — what: added the #1092 offvm-refit sizing precedent to the measured-pilot recipe (permutation-null battery at ambient 10,752-dim output: 71.9 GB RSS vs 64 planned, 12.8 h vs 5 h planned, ~2.6×; presume ≥2× until pilot-measured) — commit: 9d7efdf858 Why: session ea3a7991 (05:23Z): rf02's pilot gate aborted 4.6 h in; the plan's §9 sizing under-estimated both axes and the pilot ran only in production.

  4. Target: ~/.claude/projects/.../memory/feedback_browser_links_for_review.md — what: strengthened the browser-links memory: bare URL after EVERY writeup/dashboard edit round, not only creation (no repo commit — user-memory file) Why: Thomas had to ask for the URL 5× across two sessions today ("give me a url", "where is the new writeup", "leave it as is. Where is the URL?").

  5. filed for review #1316: widen the scripts-local lazy-import pin test scan set beyond backend_poll.py + pod_config.py (parked candidate from #1304; session spawned).

  6. filed for review #1317: code-reviewer.md Gate-scope line compliance check (parked formal candidate from #1305).

  7. filed for review #1318: fix the 4 workflow_lint no-flags FAILs on main — 3 bare Hub-verify sites + 1 unguarded upload_folder (from #1311's parked follow-up; experiment-code, wf_fix: false).

  8. filed for review #1319: torch-before-dotenv (#847 class) in issue1092_inline_figures.py + issue823_weightspace_calibration.py (from #1311's parked follow-up; wf_fix: false).

  9. filed for review #1320: batch + device-parametrize run_mlp_secondary in issue825_fit_cells.py and upload the turnstore BEFORE the fit phase (the 84-min 2×A100 wedge + lost extraction; wf_fix: false).

  10. filed for review #1321: SKILL.md same-issue follow-up loop — a mid-round defer/teardown must re-park the parent in the same action (#825 sat at followups_running ~3.5 h).

  11. filed for review #1322: CLAUDE.md orchestrator bg-discipline — ownership check before resuming/launching an agent-owned run (2 double-writer incidents) + never double-background (nohup & inside a bg Bash → false completion signal).

  12. filed for review #1323: CLAUDE.md inline carve-out — honor an explicit user "inline" override on GPU-needing follow-ups; 0-GPU "as followups" asks default inline; inline rounds commit their eval JSONs the same turn (the day's 5+ corrections + the #779 8-day-stale report).

  13. filed for review #1324: default-on env-gated vLLM hang mitigations for real-user-corpus generation in create_vllm_engine (35 min × 8 H100 lost to the known hang; wf_fix: false).

  14. filed for review #1325: pod-side-reporting push-verify must assert eval_results/ + figures/ landed in the pushed tree (2 upload-verifier round-1 FAILs today).

  15. filed for review #1326: Step-0 single-orchestrator guard is blind to unregistered inline-chat drivers (duplicate /issue 952 ran 8 min into a live round).

  16. filed for review #1327: spawn_session.py unregister subcommand (the #952 yield path hand-deleted watcher-owned registry state).

  17. filed for review #1328: CLAUDE.md ad-hoc summaries — verify numbers / head-to-head winners / "never-run" claims against the underlying artifact before asserting in chat (4+ user-corrected overclaims today).

  18. filed for review #1329: dispatch-time lint for lane-specific env vars ($WORKLOAD_ROOT) in --workload-cmd strings (the #825 RunPod-failover crash; low-confidence, planner may deflect).

  19. filed for review #1330: SKILL.md — post markers from the subagent's durable output file, never the HTML-escaped notification body (2 sessions re-derived the recipe today; session spawned).

Other problems & notes

  • Held (route 3, needs you): #1331 — pod-779's stopped volume holds the only copy of the 07-06 n10k round's per-token captures + raw rollout text; resume is host-pinned and fails on supply constraint. Carve-out: destructive/irreversible data + spend. Options in the task body: rescue-watch, accept regeneration (n50k supersedes the curve), or terminate + write off. Suggested: if the n1M round proceeds, accept/terminate.
  • Per-turn mapping figure presented with three artifacts (wrong layer, tautological prefix arm, inert shuffled null); Thomas had to push back twice before the audit caught all three — nearly entered the mentor writeup (session 09f28ede, 05:45Z). Judgment discipline: verify a null differs from the real series before presenting; partially covered by #1328.
  • Stale "rotation" claim from an orchestrator brief reached the live dashboard ~2 h before the deciding analysis landed and refuted it (teammate caught it). Judgment: mark in-flight claims provisional when briefing artifact-writers.
  • Orchestrator sent teammate directives on stale premises twice (re-assigning already-done tasks; a fix whose premise didn't hold for #1310) — teammates verified and declined; no damage.
  • #1090 fu4 was dispatched 36 s before Thomas's scope correction landed (drop sycophancy) — one wasted spawn, one duplicate --auto dispatch correctly suppressed by the guard. Judgment: let a scope exchange settle before dispatching.
  • #952 writeup needed 3 clarification rounds on refit-vs-transfer and on/off-policy framing before the numbers were labeled by fit regime; the china-politics category drop (never survived keep gates) surfaced only on interrogation despite being a body caveat. Partially covered by #1328.
  • Codex org quota exhausted until 2026-08-06 — all ~20+ review rounds today ran single-Claude (fallback machinery + #1204 pre-spawn check worked as designed; zero cross-family diversity for ~3 more weeks). Existing held task #1140 covers the pay-or-ride decision.
  • One spurious Usage Policy refusal killed one turn in the #1313 session (529/retry-dense title text); self-recovered next turn. One teammate implementer killed by repeated API 529s; durable-writes-first respawn worked.
  • Repo-root push contention ×3 during the morning merge wave (ahead=51 at one point); sync_repo_root.py recovered all three per the standing contract.
  • #1310's CPU-only fit battery held a 2×A100 GCP instance idle for hours (~30 GB un-checkpointable in-RAM state made an off-instance move worse); disposed continue-as-is in-session — its live session should add mid-battery checkpointing to issue1310_fit.py if a v4 round happens.
  • Upload-verifier round-1 FAILs in both /issue sessions (#928 eval JSONs/figures not git-committed; #1090 judge-raw VM-local only) — gate caught both; driver-side fix filed as #1325.
  • Three plan-v1 defects caught by the single-Claude review stack (#1307 misquote, #1313 must-fix, #1304 over-broad exemption) — normal gate operation, logged for calibration with Codex twins absent.
  • Minor: teammate idle-notification turn tax (~5 wakes in one session, harness behavior); one IAP-tunnel auth flake (direct SSH fallback worked); a killed 10-day-old Monitor notification enqueued into a dormant session; ~8 trivial self-recovered one-liner errors (jq/KeyError/heredoc) across sessions.
  • retraction re-check: 3 marker-mined problems scanned (the parked candidates on #1304 / #1305 / #1311), 0 skipped — no upstream retractions; transcript-mined items were tail-checked by the full-window miners (in-session fixes noted and not re-filed: #825 span crash 65ff2426a8, #779 nested-CV pooling bug, #1092 RSS-knob fix + battery-filter refit, #1314 already in-pipeline).
  • title-sync drift unchanged (8 rows — see logs/daily/2026-07-13.md).

Living-docs drift

  • living_docs.py check FAILs with 199 findings — normalized drift hash 078285c14444, IDENTICAL to the still-open 2026-07-07 re-synthesis proposal (~75 dead legacy anchors, relates_to ⇄ evidence mismatches incl. #722/#763/#494–#501, ~54 unlinked completed clean-results, 7 stale spec-role-header State-trailer rows). docs/open_questions.md unchanged since 2026-07-07.
  • drift proposal already open (logged 2026-07-07, extending 06-29/07-04); not re-proposing.
  • Parked-proposal pass: 1 un-reconciled living-docs rejection (#713, 2×) — context unchanged, not re-surfaced; 0 parked-redundant follow-ups eligible for revival.

My thoughts

Highlighted results

  • no results promoted today