EPS Dashboard

What happened

Heavy autonomous-/issue day, mostly machine-driven. Five experiments running at day's end (#501, #503, #506, #507, #508), three parked blocked awaiting a Thomas decision (#504, #505, #509). Three tasks reached awaiting_promotion overnight/early (#489 @02:00Z, #498 @05:49Z, #502 @10:25Z) but none were promoted to completed — promotion queue grew, nothing cleared.
#506 (FWFT marker on a 27B model): planned + 4 code-review rounds against Qwen/Qwen3.5-27B, which turned out to be a video-language multimodal model transformers can't load as a causal LM — crashed all 3 arms at first launch on an 8×H200. Thomas redirected to Qwen3-32B ("Why arent we using qwen 2.5 7b"); relaunched, then hit a silent gate-parse bug + missing flash_attn + a MooseFS save crash. FWFT arm still training at session end.
#501 (multi-turn drift supplement): two vLLM CUDA OOMs at Phase-4 prompt-logprobs (20:37Z, 21:26Z), relaunched twice; Phase-1 JS sweep ran ~16h vs a ~4h plan estimate. Sweep healthy at ~71/288 cells by 06:24Z.
RunPod $80/hr account cap was the dominant blocker: #507, #508, #509 each ran their entire plan+review pipeline and then stalled at pod provisioning because four pods (#501/#503/#504/#506) already saturated the cap. Zero GPU wasted, but full pipelines stranded at the last step.
#503 escalated to Thomas after 4 launch attempts surfaced 5 cumulative upstream/training gaps (the sweep driver had no training phase at all); he picked "add the missing training step," implementer reached round 6-8, relaunched 01:47Z.
PM session applied 5 of 6 of yesterday's /daily proposals (commit cluster this session): RunPod spend-guard (pod_lifecycle.py), planner config-load smoke (planner.md §11), autonomous-session text-menu ban (issue/SKILL.md), session-liveness cross-check (experiment-status.md + stuck-diagnoser.md), and the merge fix. Proposal #1 (data-gen-via-Haiku rule) was left as historical record by choice. A CLAUDE.md "reuse existing experiment code" Critical Rule was also added + pushed (d17329b06).

Proposed workflow improvements

Note: yesterday's proposals #2-#6 were already applied in today's PM session (spend-guard, planner config-load smoke, autonomous text-menu ban, session-liveness cross-check). The proposals below are the new / residual gaps surfaced by today's transcripts. Ordered by leverage.

Target: .claude/agents/experiment-implementer.md + .claude/skills/issue/SKILL.md (Step 6d.0-bis smoke gate) — what: require a real pre-pod end-to-end dry-run, not a fixture/stand-in smoke. This is the single highest-leverage fix — the SAME class of bug burned pod launches in #501, #502, #503, AND #506 today because CPU smokes used canned data / stand-in models / read config.json directly. Why: #503's driver had no training phase ("GAP 5: training step missing from driver entirely … each launch closes a gap, surfaces another", 4 launches); #502 crashed on Class-D rewrites the smoke never exercised; #501's Phase-4 --smoke "writes hardcoded delta_g=2.0 … no vLLM, no LoRA" and a real count(MARKER_ID)==1 crash was invisible to it; #506 was planned + reviewed 4× against a model nobody ever AutoConfig.from_pretrained'd. Proposed edit:

+ ## Mandatory pre-pod end-to-end dry-run (before the FIRST pod launch)
+ A `--smoke` path must exercise the REAL code path, not fabricated constants. Before posting
+ epm:proposed-tests / passing code-review, assert the smoke:
+   (a) does a real AutoConfig.from_pretrained(model_id) + architecture-class check (causal-LM, not multimodal/video);
+   (b) verifies every artifact the run later CONSUMES (adapters, R_train/R_eval, panels, centroids) has a
+       producing phase in THIS driver, or cites where it was already produced (+ that it is on `main`);
+   (c) runs >=1 cell of EVERY condition class the production run will execute (e.g. Class-D), at production
+       max_model_len / gpu_memory_utilization so VRAM-spike (prompt_logprobs) bugs surface at smoke;
+   (d) checks every per-arm dependency is installed (e.g. FWFT needs flash_attn).
+ Fabricated-constant smoke stubs (hardcoded delta_g, "Smoke sample 1") are forbidden.

Target: .claude/skills/issue/SKILL.md (Step 9b / 10d merge) + CLAUDE.md — what: enforce that parent-experiment shared-infra code merges to main at clean-result promotion, not just the body. Why: #502's dispatcher + round-4/5/7/8 bug fixes (issue502_dispatch.py, issue502_generate_probes.py) stranded on the issue-502 branch blocked both its children today — #509 had to "surgical-checkout a stranded file at SHA 0ab4d16f", and #511 parked at the clarify gate because it would "re-trip the same KeyError: 'extraction_point' #502 already fixed." #494 also left PR #414 dangling and its branch 318 commits behind. Proposed edit:
```
+ At promotion/merge time, any NON-experiment-specific code the run added under scripts/ or src/
+ (shared dispatchers, prep scripts, bug fixes to reused infra) MUST land on `main`. A follow-up/child
+ task may not be left to cherry-pick a fix stranded on a parent's experiment branch. If the full rebase
+ is deferred (Guard 3), the shared-infra files specifically are still cherry-picked to `main`.
```
Target: .claude/agents/experiment-implementer.md + .claude/agents/code-reviewer.md — what: when reusing a sibling's dispatcher, assert every recipe constant is actually threaded and every cached keyed artifact covers the chosen key set. Why: #504 and #505 both inherited the #472 dispatcher and hit the SAME root across ~13 / ~6 rounds: recipe constants silently shadowed by the sibling's defaults (#505: LORA_R=16 set in constants but train_one_cell defaulted to rank 32 → vLLM crash; "three distinct dispatcher recipe-threading bugs share one shadow-constant root cause"), and cached R_train.json/R_eval.json that didn't cover the newly-picked personas (KeyError 'architect' shipped to launch because both reviewers reasoned panel ⊆ bank ⇒ panel ⊆ R_eval.keys()). A workflow-fix-candidate for this was emitted in-session. Proposed edit:
```
+ ## Reusing a sibling dispatcher
+ + Assert every recipe constant (lora_r, alpha, lr, epochs, persona set) is threaded into the inherited
+   train_one_cell / eval call — not silently shadowed by the sibling's default.
+ + When consuming a cached KEYED artifact (R_train, R_eval, centroids), verify the consumed key set is a
+   subset of the artifact's ACTUAL keys, not just of the persona bank. Auto-fill missing keys (train AND
+   eval) before launch.
```
(code-reviewer gets the mirror lens: "cached-artifact-coverage — a diff consuming a keyed artifact must verify key-set coverage against the artifact, not a superset.")
Target: .claude/agents/critic.md + .claude/agents/code-reviewer.md + .claude/agents/experiment-implementer.md — what: (a) a driver must produce every artifact it consumes; (b) the run driver must be a committed, reviewed repo script, never generated on the pod at launch; (c) gate parsers must raise on a missing key, never fall back to a sentinel that reads as a real outcome. Why: #503's driver consumed adapters it never trained. #506's wrapper "lives only on the pod (created at launch)" — which is structurally WHY its gate-parse bug escaped all 4 review rounds: it parsed cells instead of cell_summaries, fell back to -1.0, reported stage0_pass=0, and silently skipped Phase 2 for both LoRA arms even though the install actually passed (a fail-fast violation). Proposed edit:
```
+ critic/code-reviewer lens: a sweep/eval driver that LOADS an artifact (adapter, panel, R, centroid) must
+   contain the phase that PRODUCES it, or cite the producing task + that the artifact is on `main`.
+ experiment-implementer: the production run driver is a committed repo script reviewed in the diff — do NOT
+   write/generate the driver on the pod at launch. Gate parsers raise on a missing expected JSON key; never
+   fall back to a sentinel value that is indistinguishable from a legitimate experimental outcome.
```
Target: .claude/rules/persona-distance-metrics.md — what: mandate a single canonical persona-distance cosine operationalization (globally mean-center the centroid bank, then L2-normalize, then cosine). This is also today's biggest research-integrity flag. Why: early canonical leakage tasks (#66/#77/#99/#228/#311/#380) globally mean-center + L2-normalize before cosine; mid/late tasks (#405/#406/#460/#472/#474/#477/#478/#490 and #504 r1-5) skip mean-centering, compressing the geometry ~6×. Same recipe → ~10-20× different absolute ΔG. Several MODERATE/LOW leakage-predictor calls (esp. open-q 3.4a) are now reanalysis candidates. Proposed edit:
```
+ Canonical persona-distance cosine: globally mean-center the centroid bank, THEN L2-normalize, THEN cosine.
+ New predictor code MUST assert the mean-centering step ran. Tasks that used raw (un-centered) cosine
+ (#405/#406/#460/#472/#474/#477/#478/#490/#504-early) are NON-COMPARABLE to the mean-centered line and any
+ cross-task ranking claim drawn across the two regimes is a reanalysis candidate.
```
Target: .claude/skills/issue-tick/SKILL.md (+ scripts/spawn_session.py autonomous watcher) — what: (a) a /loop-driven or autonomous /issue must stop re-invoking once the task is at a terminal/park state; (b) detect a "Not logged in" / auth-expired tick and halt + PushNotification instead of firing into a dead session. Why: the /loop 10m /issue 494 session fired 160 no-op /issue 494 invocations after the task parked at awaiting_promotion, and from ~22:32 PT onward 121 consecutive ticks returned 'Not logged in · Please run /login' — ~20h of schedule burned silently with zero tool calls and no alert. Proposed edit:
```
+ If task status is already awaiting_promotion/completed/archived, the tick emits "parked — nothing to do,
+   stopping loop" and tears down the cron / signals the loop driver to stop, instead of re-running cold.
+ If a tick's assistant turn is an auth/login error ("Not logged in", "Please run /login"), fire
+   PushNotification and HALT the loop — do not keep re-firing into a logged-out session.
```
Target: CLAUDE.md (Routing experiment intent table) + .claude/skills/issue/SKILL.md — what: define how a kind: analysis task that needs new code + a pod run is routed. Why: #509 is kind: analysis but its plan needed ~300 LOC + an 8×H100 run; the strict analysis → analyzer (re-analysis only) routing didn't fit, so the orchestrator silently improvised into the experiment-implementer/pod path on its own judgment (undocumented). Proposed edit:
```
+ An `analysis` task that requires NEW code or a pod run is dispatched through the experiment-implementer /
+ pod path (Step 4+), while keeping the §11 / measurement-validity exemptions of `analysis`. The clarifier
+ flags an analysis task whose plan implies a pod run so the routing is explicit, not improvised.
```
Target: .claude/skills/issue/SKILL.md (Step 6b) + scripts/spawn_session.py — what: (a) admission-control / sequence autonomous spawns against the RunPod cap so they don't each do hours of planning then stall at provision; (b) pick ONE canonical park state for "provision refused, capacity wait." Why: #507/#508/#509 each completed a full plan+review pipeline before discovering at Step 6 that the cap was saturated. And the park state diverged: #508 stayed running with epm:pod-pending, #509 went blocked for the identical cause. (The spend-guard added today refuses early at the pod level but nothing sequences the spawns before they burn planning effort.) Proposed edit:
```
+ Before spawning an autonomous /issue that will need a pod, check projected hourly burn vs
+ RUNPOD_ACCOUNT_HOURLY_CAP; if the new pod wouldn't fit, queue the spawn rather than letting it plan+review
+ then stall at Step 6. On a provision-refused-for-capacity outcome, use ONE canonical state (recommend:
+ status:blocked + epm:pod-pending v1) so the dashboard + re-invocation logic are consistent.
```
Target: .claude/rules/code-style.md (checkpoint-per-phase) + .claude/skills/issue/SKILL.md (compute-deviation watchdog) — what: every multi-phase launcher ships --resume + skip-if-done guards at plan time; raise the watchdog at the first 2× over-budget projection. Why: #501's launcher had no --resume — after Phase-4 OOM'd, restarting risked re-burning the 16h Phase-1 sweep, so resume had to be patched mid-recovery; and the plan's ~4.1h estimate vs the ~16h+ actual (≈4× under) wasn't caught until hour 16, despite the autonomous cap auto-approving on the wrong estimate. Proposed edit:
```
+ Multi-phase launchers ship --resume + skip-if-done (idempotent per cell/phase) at plan time, not as a
+   recovery patch. Smoke artifacts write to a namespaced dir so they never poison a production --resume.
+ Compute-deviation watchdog fires when projected wall time exceeds 2x the plan estimate (extrapolated from
+   the smoke's per-unit timing), not only at terminal over-run.
```
Target: .claude/skills/issue/SKILL.md (Steps 2b, 5c) + CLAUDE.md (cap-3 rule) — what: four small ensemble-review integrity guards. Why: #508 — Step 2b consistency-checker was skipped before epm:plan and back-filled "as a safety net." #502 — round-3 was Claude PASS / Codex FAIL with no reconciler ("irregularity"), carried into round 4. #503 — round-3 verdict "was never posted (orchestrator went silent)", forcing a resumed session to reconstruct state. #502/#506 — round counter hit 4 on a "pragmatic call"; unclear whether a NEW post-launch bug resets the per-reviewer cap-3. Proposed edit:
```
+ Step 2b: consistency-checker marker is a HARD precondition of posting epm:plan (assert it exists first).
+ Step 5c: a PASS-vs-FAIL ensemble round cannot advance without a reconciler verdict marker; always post a
+   verdict marker (even FAIL/INDETERMINATE) before any status transition.
+ cap-3: a NEW substantive bug surfacing post-launch RESETS the per-reviewer round counter (failure-driven),
+   distinct from the same blocker cycling — make the reset explicit so it isn't a per-case judgment call.
```
Target: .claude/agents/planner.md (§9 sizing, §11 cost) + scripts/pod.py GPU-intent table — what: fix the GPU-h cost model and pod sizing. Why: #505's first estimate was off ~200× ("120 GPU-h" to triple epochs, corrected to ~0.6 because eval, not training, dominates wall-time). #507's plan claimed 4×H200 gives ~1.4× per-GPU throughput (actually ~1.05-1.08×) and mis-counted LoRA optimizer state. #509 specced an 8×H100 (inf-70b) pod for a no-training Qwen-2.5-7B forward-pass bake-off ("fits with massive headroom") — exactly what tipped the account over the cap. Proposed edit:
```
+ Cost estimates separate training-compute from eval-compute; never scale total wall-time linearly with
+   optimizer steps. Pin per-GPU throughput-scaling + LoRA-vs-full optimizer-state memory to grounded sources.
+ Pod size = smallest spec that fits the model for the job; an inference/bake-off on a 7B model defaults to
+   1x H100, NOT an inherited inf-70b 8-GPU spec.
```
Target: .claude/skills/issue/SKILL.md (worktree handling) + src/explore_persona_space/orchestrate/preflight.py — what: stop concurrent-session task.py event commits from poisoning reviewer diffs / preflight / merges. Why: worktree branches ran hundreds of commits behind main purely from concurrent task-state commits (#502: 859 behind; #503: 274→16; #506: 121 behind → preflight bypassed). This produced reviewer-diff pollution (72 foreign tasks/ paths), a Codex false-positive "event-ledger deletions" Critical, and stranded merges. Proposed edit:
```
+ Rebase the worktree branch onto origin/main at run-launch (or warn) so the end-of-pipeline merge isn't
+   blocked by hundreds of concurrent task.py commits.
+ Preflight: classify "N commits behind origin/main" as PASS-with-warning when the delta is task-state-only
+   (no src/ configs/ scripts/ changes), instead of forcing a manual bypass decision.
```
Target: .claude/rules/marker-leakage-measurement.md (eval-guard constants) — what: make the assert_adapter_actually_applied threshold geometry-relative, not an absolute nat value. Why: #504 parked blocked because the eps=0.50-nat eval-guard (inherited from #477's raw-cosine anchor) false-trips on a legitimately-trained adapter under mean-centered geometry, where the same recipe yields 10-20× smaller absolute ΔG. Proposed edit:
```
+ The adapter-applied eval-guard eps scales to the run's OWN source-self ΔG (geometry-relative), not an
+ absolute nat threshold carried across centering regimes.
```
Target: scripts/poll_pipeline.py + scripts/pod.py — what: stop false "dead" verdicts and reap stale-EXITED pods at provision. Why: #502 — a clean 171s run was marked "dead" because the PID was gone (clean exit indistinguishable from crash); a transient SSH-resolution failure was also marked "dead". #506 — stall-check emitted a 1000000000 sentinel because its log glob didn't match the per-arm filename. #509 — two stale-EXITED pods (pod-505, pod-489) counted against capacity and were only "flagged," not reaped. Proposed edit:
```
+ Poller checks for a completion sentinel / rc=0 before classifying PID-gone as dead; re-runs `pod.py config
+   --sync` + retries once before a "dead" verdict on an SSH-resolution failure; fix stall-check log glob.
+ Provision-time capacity check auto-terminates EXITED pods (or surfaces them as reclaimable $/hr).
```
Target: .claude/agents/experiment-implementer.md + .claude/skills/issue/SKILL.md (Step 5/6 ordering) — what: (a) advisory/equivalence gates must warn, not exit-nonzero under set -e and kill downstream phases; (b) clarify how the GPU-dependent smoke gate is satisfied when the pod is provisioned in Step 6 but the gate is checked in Step 5. Why: #494 — an in-script "byte-identical-within-1e-4" consistency gate hit expected float noise, returned rc=2 under set -e, and aborted Phases 2+3 before they launched (Phase 1 results were valid); and the Step-5 smoke gate demanded a GPU run before the Step-6 pod existed (nvidia-smi: command not found), forcing an out-of-band provision. Proposed edit:
```
+ Advisory / recipe-equivalence checks WARN + record a caveat; they never exit-nonzero under `set -e` in a
+   chain that gates subsequent phases. (Note: the "byte-identical" framing is itself banned per the
+   clean-result spec — use a tolerance-band check.)
+ For GPU-dependent experiments, the Step-5 smoke gate may be satisfied by a pod provisioned at/before
+   code-review, OR `smoke-run-missing` defers the GPU smoke to the Step 6d.0 parity check.
```

My thoughts

Highlighted results

no results promoted today

Daily — 2026-06-06 (5 autonomous experiments in flight; RunPod cap + smoke-coverage gaps dominate; 0 promoted)

What happened

Proposed workflow improvements

Other problems & notes

My thoughts

Highlighted results