EPS Dashboard

What happened

#448 dominated the day (70 commits). The user halted the off-policy version mid-flight ("STOP — do not promote #448, its DV is off-policy") and forced a full re-run with the on-policy design. The 12-cell contrastive-LoRA sweep landed a clean NULL headline: none of four recipe knobs moves on-policy bystander marker leakage at this anchor's training budget, where the on-policy log-prob has saturated. Reached awaiting_promotion (MODERATE).
#458 checkpoint-deletion post-mortem + clean-result. Root-caused the data-loss incident (delete-after-eval rm -rf racing a best-effort WandB/HF upload under MooseFS + HF quota pressure → 20 of 36 trained cells finished with no recoverable artifact). Shipped the fix bundle: trainer.py fail-loud adapter-persist before the rm (merged to main 04265e50a), a CLAUDE.md Upload Policy rule, and the guard_repo_root_branch.sh PreToolUse hook. Clean-result reached awaiting_promotion (cosine similarity = coarse code-vs-prose detector, not a per-dataset EM predictor; MODERATE).
Three other experiments advanced. #459 (turner_em retrain) plan v2 approved → code-review r1 CONCERNS → r2 PASS → running. #462 (epoch-resolved on-policy marker transfer) ran train→crosseval→done. #463 (re-run EM predictors with full-response Rao-Blackwellized KL/JS) created + launched. #461 produced the predictor scatter grid + a mechanism correction.
Workflow + infra landings. CLAUDE.md gained canonical KL/JS-divergence + cosine-similarity definitions (full-response Rao-Blackwellized; persona-vectors cosine) after the user corrected a first-token-only JS implementation. Two workflow-fixes auto-applied (#448 experimenter pre-launch flag check; #408 eval-smoke-gate extension). Dashboard /updates/<N> → /tasks/<N> redirect deployed, fixing the Dan/Christina mentor-link 404s from the May-29 refactor.
Day ended on a snag. #444 hit a preflight import-deadlock (last commit 22:12 "partial progress"); a 4×H100 pod had already idled ~21h on an unfired human gate. 0 clean-results promoted to completed during the June 1 PT day — #448 and #458 are parked at awaiting_promotion awaiting Thomas's manual task.py promote.

Proposed workflow improvements

Ordered by severity. PROPOSED ONLY — reply "do 1, 4, 6" (or "do all") and workflow-improver applies the greenlit ones. Script-side fixes (poll_pipeline.py, pod.py, preflight.py, bootstrap_pod.sh) are out of /daily proposal scope and live in ## Other problems & notes below.

Target: .claude/skills/issue/SKILL.md — what: stop/terminate the pod when the pipeline blocks on a human gate OR crashes, before the session exits. Why: idle-pod burn recurred three times today, ~ $1k total. #444 idled a 4\timesH100 ~21h on an unfired `epm:fact-pick` gate (session dc2e0689: "roughly$ 170-250 of idle burn"); #404 idled ~2 days because Step 8 auto-terminate never fired after the pipeline skipped upload-verification (session dc93f209: "$600-800 already spent doing nothing"); #407 crashed in the aggregate phase and left a 4×H100 idle ~1 day (session 404cb6d9). The orchestrator leaves GPUs billing whenever a run dies or blocks outside the normal Step 8 path. Proposed edit:
```
+ ### Pod must not idle on a halt
+ When the pipeline (a) blocks on a human-input gate that cannot be satisfied
+ this turn (e.g. epm:fact-pick) or (b) is detected crashed/dead with GPUs idle,
+ run `pod.py stop --issue <N>` (volume preserved; resume re-provisions) or
+ `terminate` BEFORE exiting the turn. Never leave a pod RUNNING while awaiting
+ human input or after a crash. (Incident 2026-06-01: #444/#404/#407, ~$1k idle.)
```
Target: .claude/agents/experiment-implementer.md — what: when porting a recipe from an unmerged parent branch, diff and port the WHOLE train+eval code path up front instead of cherry-picking function-by-function. Why: #451 cherry-picked factor_screen_397 but left train/sft.py at main's older TrainLoraConfig signature → all 72 cells crashed in ~10 min ("the cherry-pick brought the caller but not the callee change"). #456 hit the same class three times ("third partial-port — cherry-picking individual functions keeps missing siblings"), each crash burning a fix-relaunch on a live pod. Proposed edit:
```
+ ## Porting a recipe from an unmerged parent branch
+ If the parent's scripts/configs live on a branch never merged to main
+ (e.g. issue-432 @ <sha>), do NOT cherry-pick functions one at a time.
+ Run `git diff <parent-sha> -- <full train+eval path>` and port/verify the
+ ENTIRE divergence in one cycle before launch. Partial ports crash the pod one
+ phase at a time (#451: 72 cells; #456: 3 partial-port crashes, 2026-06-01).
```
Target: .claude/skills/issue/SKILL.md — what: require a tiny-N end-to-end smoke run (every phase: gen → drift → train → eval → aggregate) before any multi-phase data-gen production launch; FAIL blocks production. Why: #408 hit 5+ distinct data-gen bugs discovered one-per-pod-cycle over ~~41h idle (~~$103 bleed) because the resume-skip design serialized bug discovery and no fast smoke run exercised every phase first (session e1e53b47: "the data-gen pipeline was never smoke-tested end-to-end before the expensive production run"). The #408 eval-smoke-gate extension landed today covers the eval cell, not the full gen→train→eval pipeline. Proposed edit:
```
+ ### Pre-launch end-to-end smoke gate (multi-phase data-gen pipelines)
+ Before the GPU production launch, run the FULL pipeline at tiny N (≈2-3 rows,
+ 1 cell, 1 seed) so EVERY phase executes once on CPU/1-GPU. A resume-skip design
+ hides downstream bugs; the smoke run forces them all up front, in minutes not
+ one-per-pod-cycle. FAIL blocks production. (Origin: #408, 5+ serialized crashes.)
```
Target: .claude/skills/issue/SKILL.md (Step 10) — what: merge guard + status-flip ordering so a behind-main branch can't revert another task's state and a crash mid-merge can't strand a task at running. Why: the #458 merge branch (1,146 commits behind main) silently changed tasks/running/448/events.jsonl — another task's state (session dc93f209); the assistant caught it by hand and git checkout origin/main'd the foreign file. Separately #458's folder said running after the work was done, pod terminated, and branch merged — the session died between merge and status flip (session dc2e0689: "status drift — folder still says running"). Proposed edit:
```
+ ### Step 10 merge safety
+ 1. `git diff --name-only origin/main HEAD -- tasks/` MUST be empty except this
+    task's folder. Any foreign tasks/ file in the merge → `git checkout
+    origin/main -- <that file>` before push. Never let a behind-main branch
+    revert another task's events.jsonl.
+ 2. Flip status OFF `running` BEFORE merge-to-main + pod terminate, so a crash
+    mid-step can't strand a terminated-pod task at `running`. On `/issue <N>`
+    resume: if pod terminated AND branch merged AND status still `running`,
+    auto-advance instead of treating it as in-flight.
```
Target: .claude/agents/interpretation-critic.md (Lens 1, measurement validity) — what: hard-FAIL a marker-leakage DV that reads log p(marker) teacher-forced at a fixed position after a canned/fixed response the model didn't generate. Why: #448 reached awaiting_promotion measuring the marker off-policy — the exact #432→#456 anti-pattern that CLAUDE.md forbids — and only the user's manual "STOP" caught it (session 49bae5c4). The measurement-validity lens should block it before promotion-readiness, not rely on Thomas. Proposed edit:
```
+ For marker-leakage DVs: hard FAIL (not a concern) if the eval reads
+ log p(marker) at a fixed position after a CANNED/teacher-forced response the
+ model did not itself generate. The marker MUST be measured on-policy, after
+ the model's own generation. This is the #432→#456→#448 anti-pattern and is a
+ hard block per CLAUDE.md.
```
Target: ~/explore-persona-space/CLAUDE.md (Upload Policy) — what: replace the dead hf api list-repo-files command with the valid hf repo-files list. Why: the installed hf CLI has no api subcommand; it errors to stderr and | grep swallows it as an empty result that reads as a false "0 files uploaded." In the #458 post-mortem this nearly produced a wrong "checkpoints don't exist" conclusion (session 17478122: "that earlier '0' was a CLI error, not a real result"). Proposed edit:
```
- Verify post-data-gen: `hf api list-repo-files superkaiba1/explore-persona-space-data --revision main | grep <bucket>`.
+ Verify post-data-gen: `hf repo-files list superkaiba1/explore-persona-space-data --revision main | grep <bucket>`
+   (the `hf` CLI has NO `api` subcommand — it errors to stderr and `| grep` reads it as an empty/zero result; #458 post-mortem nearly drew a wrong conclusion from the silent "0").
```
Target: .claude/skills/pm/SKILL.md + .claude/skills/daily/SKILL.md — what: prepend export PATH="$HOME/.local/bin:$PATH" and use uv run python in the opening scan commands. Why: non-login shells lack ~/.local/bin on PATH and there's no bare python. The /pm triage scan failed python: command not found (session dc2e0689); the /daily first parallel batch failed command not found: uv and even cancelled a sibling parallel call (session 0a149a32) — and this same failure hit me again at the start of this run. Three strikes today. Proposed edit:
```
- python scripts/task.py list-by-status --limit 500
+ export PATH="$HOME/.local/bin:$PATH"   # uv lives in ~/.local/bin; non-login shells miss it
+ uv run python scripts/task.py list-by-status --limit 500
```
Target: .claude/agents/analyzer.md — what: add a pre-emission register self-check so first-draft clean-result bodies pass Lens 2/7/13 without a critic bounce. Why: the analyzer's first drafts repeatedly tripped the long-standing rules — #408 critic "FAIL — Lens 2 (opaque condition codes) and Lens 7 (named tests + bracketed-CI bounds in narrative prose) ... an AI/process tell ('the codex critic surfaced') + shouty caps" (session e1e53b47); #458 needed a Lens-13 coverage fix (session dc93f209). Each costs a REVISE round. Proposed edit:
```
+ ## Pre-emission register self-check (run before posting the draft body)
+ - [ ] No opaque condition codes (B@k, A, M1, cond_4) in TL;DR/Findings prose — plain-English names only
+ - [ ] No named statistical tests / bracketed CIs in narrative prose (Reproducibility only)
+ - [ ] No process/AI tells ("the codex critic surfaced") or shouty CAPS
+ - [ ] `### What I ran` flags any planned cell/seed/factor that silently dropped (Lens 13)
```
Target: .claude/agents/clean-result-critic.md — what: add a lens that hard-FAILs an H1 title / TL;DR headline resting on an arm the body itself flags as contaminated or failed-its-own-data-gate. Why: the #407 clean-result was titled/framed "content-agnostic gating" off an arm with contaminated training data (stale paraphrases) and string-lookup-inflated multiple-choice numbers; the user had to interrogate it repeatedly ("how did taught-wrong-info get ~100%?" / "mark it as bugged") before it was demoted (session 404cb6d9). Proposed edit:
```
+ Lens: if the body discloses a data-validity failure (contaminated training
+ pool, a Phase-0/K1 gate the arm failed, wrong base prior) for an arm, the H1
+ title and TL;DR headline MUST NOT rest a positive claim on that arm. Hard FAIL
+ — demote to a surviving clean arm or to "bugged."
```
Target: .claude/skills/issue/SKILL.md — what: require a tasks/ + HF/WandB sweep for the answer before any non-gate AskUserQuestion. Why: the orchestrator twice surfaced a design-fork question before exhausting the resourceful path and was rejected both times — "Look deeper at other issues" after a reuse-vs-retrain ask (session c609f6f5); "Show me the body first" after a title-options ask (session 404cb6d9). Reinforces existing halt-criterion #1. Proposed edit:
```
+ Before any non-gate AskUserQuestion (design forks like reuse-vs-retrain, title
+ options), FIRST sweep tasks/ + HF/WandB for artifacts/answers that resolve it.
+ Ask only once the investigation leaves a genuine gap only the user can fill.
+ ("Look deeper first" / "show me the body first" rejections, 2026-06-01.)
```
Target: ~/explore-persona-space/CLAUDE.md (the repo-root-branch rule) — what: add a post-commit verification step for mid-session workflow-file edits. Why: the corrected on-policy marker-leakage rule was committed but stranded on fix/sweep-ckpt-persist by a concurrent branch switch/merge, leaving the WRONG first-token version live on main until the user happened to ask "did you add this rule to CLAUDE.md?" (session 8e691b41). The guard_repo_root_branch.sh hook shipped today prevents the branch-switch cause but not a merge sweeping a committed edit onto a branch. Proposed edit:
```
+ After any CLAUDE.md / workflow-file edit committed mid-session, immediately
+ `git log -1 --oneline -- <file>` on `main` to confirm the commit landed on main
+ and was not swept onto a concurrent feature branch (incident 2026-06-01: the
+ corrected marker-leakage rule stranded on fix/sweep-ckpt-persist; main kept the
+ wrong first-token version).
```
Target: .claude/skills/paper-plots/SKILL.md — what: require a color / stacked-bar segment to mean ONE fixed category across all panels of a figure. Why: the #407 per-framing figure put "counter" in bar position 2 of the contradictory panels but "refusal" in that position in the refusal panels, so columns weren't comparable; the user caught it ("the bars are showing different things in the graph?") (session 404cb6d9). Proposed edit:
```
+ Consistent encoding across facets: a given color or stacked-bar segment MUST
+ mean the same category in every panel. Use a fixed decomposition (e.g.
+ taught / counter / refusal / other) so columns are directly comparable.
```
Target: .claude/skills/issue/SKILL.md — what: announce a follow-up/child task in chat at creation time (id + title + parent + status). Why: #461 was filed mid-session and worked on, but never announced, so the user lost track — "What is 461?" (session 9e667517). Proposed edit:
```
+ On creating a follow-up/child task mid-session, immediately post one line in
+ chat: "Filed #<N> '<title>' (child of #<parent>, status:<s>)". Don't let a
+ created task stay invisible until the user asks what it is.
```
Target: ~/explore-persona-space/CLAUDE.md (Task Workflow API) — what: note that task.py must be invoked from repo root / by absolute path. Why: after cd "$(... task.py find <N>)", a later uv run python scripts/task.py ... resolved the relative path against the task folder → No such file or directory (session e1e53b47). Minor but recurring foot-gun. Proposed edit:
```
+ Note: after `cd "$(... task.py find <N>)"` you are in the task folder, not repo
+ root — invoke task.py by absolute path or `cd` back first; `scripts/task.py` is
+ relative to repo root.
```
Target: .claude/skills/daily/SKILL.md (Commit step) — what: use git add -f for the daily file, because logs/ is gitignored. Why: hit live in this run. The skill's git add logs/daily/YYYY-MM-DD.md silently commits nothing (logs/ is .gitignore line 70: "no changes added to commit"), so the daily never lands in git or the dashboard. The 4 prior dailies are tracked only because they were force-added; the skill text doesn't say to. Proposed edit:
```
- git add logs/daily/YYYY-MM-DD.md
+ git add -f logs/daily/YYYY-MM-DD.md   # logs/ is gitignored; force-add to match prior dailies
  git commit -m "logs: daily stub for YYYY-MM-DD"
```

My thoughts

Highlighted results

no results promoted today (June 1 PT). #448 and #458 reached awaiting_promotion (clean-results drafted, not yet promoted); #432 was promoted at 23:20 PT May 31 and belongs to the May 31 daily.