EPS Dashboard

What happened

2 clean-results promoted to completed: #390 (refusal-style negatives install a persona gate that generalises across 9 of 11 OOD eval framings, MODERATE) and #391 (could not selectively implant sycophancy on Qwen2.5-7B-Instruct — trained source + 23 bystanders move in lockstep, LOW). Both promoted 2026-05-28. ~6 more clean-result drafts landed at awaiting_promotion this week (#385, #404, #415, #416, #432, and #407 which then went back to interpreting), but only the 2 above were actually promoted — the interpret/promote backlog Dan flagged is still real (53 tasks sitting in awaiting_promotion).
Heavy experiment churn, lots of recovery. 6 tasks running at week end (#406, #408, #444, #448, #451, #452), 1 interpreting (#407), 1 planning (#392). The two hardest threads: #408 (8+ code-review rounds, 6 sequential data-gen bugs, then a genuine fixed-seq-len-vs-verbose-corpus design flaw — k≥10 turn prefixes hit ~71k tokens vs the planned 4096, truncating the marker out of 47-91% of rows → needs a re-plan, not a patch); and #407 (a cross-regime methodology blowup reframed over 3 interpretation rounds).
Big mentor-facing dashboard build + clean-result-workflow evolution (mostly 05-26 → 05-29): the /updates + /log results timeline, Sagan-style anchored comments, per-card @claude chat, a TOC + collapsible task pages, and a wholesale clean-result body restructure (mentor-facing titles, Goal→Motivation, optional Next steps, one-takeaway/one-figure). Most of the 05-26 batch landed and is logged in iterations.md; the larger #432-dictated overhaul (2026-05-31) did not propagate to SPEC.md — see proposal #1.
Large workflow-surface churn: a 10-workstream recurring-bug-fix batch (preflight disk-budget, pod_disk_guard, runpod retry+supply, bootstrap PATH, codex retry, watchdog PID, training-pipeline guards, hub artifact-verify, dashboard dep test), the living-docs machinery (Phases 1-4 + living-docs-updater), reconciler marker canonicalization to epm:review-reconcile, the MDX-safety body gate (<https autolinks + <digit, applied), the setsid SSH-safe launcher, and a ~50% CLAUDE.md trim (the >40k-char performance warning is resolved). Hyperparameter-grounding (literature + past-issue Source:) was added to planner/critic.
My Goat genesis (2026-05-25): the OpenClaw→Claude Code personal-assistant migration session — one-time setup, not project friction.
Pod money-leaks were the week's most expensive recurring class: #406 orphan 2×H100 idle ~24h, #382 leaked 1×H100 ~3 days (Step 8 skipped), pod-397 idle, pod-407 silently-failed launch sitting idle ~~3 days (0 GPU, no log), and #382 ran out of money mid-run with no checkpoints (~~$215 redo).

Proposed workflow improvements

Severity-ordered. #1 and #3 are merge/greenlight decisions, not one-shot workflow-improver edits — flagged inline. Reply e.g. "do 2, 4, 5" / "do all" / "skip 1 — let's discuss".

Target: .claude/skills/clean-results/SPEC.md (+ propagate to .claude/agents/analyzer.md, .claude/agents/clean-result-critic.md, scripts/verify_task_body.py) — what: adopt the clean-result body overhaul Thomas dictated against #432; SPEC.md was never touched and is now in large spec-vs-practice drift. LARGE / architectural — needs your explicit go-ahead and careful multi-file propagation; do not auto-apply. Why: verbatim (session ad406932): "There should be no separate 'details' section. The parameters should go into reproducibility. Methodology corrections is not necessary... Basically it should just be: # Human TLDR (Empty) # TLDR Lesswrong style description... separated into different subheadlines with the result, one plot per result, with one example of raw completion per artifact, dropdown of more raw completions, and links to all the raw completions. Next steps shouldn't be there by default." Plus: "Remove why confidence is where it is"; "training data should be a drop down of 5 examples... and then a link to the full thing. Same for eval data"; "The TLDR should start with the motivation and it should be labeled as motivation"; "Every clean result, except in the motivation, should stand alone... Never use the phrase byte identical." He explicitly said "eventually we'll fix the entire clean result workflow so it follows all this." (Absorbs the 05-26 "Human TL;DR required at top" item.) Proposed edit:
```
# SPEC.md — collapse 4 required H2s to two top-level sections
- ## Human TL;DR / ## TL;DR / ## Details / ## Reproducibility
+ ## Human TL;DR  (empty; Thomas authors/approves before promotion)
+ ## TL;DR  (LessWrong narrative: Motivation -> What I ran -> per-result H3
+   subheads; one plot per result; ONE relevant raw completion + a details
+   dropdown + full link; raw data co-located with its section — training
+   data near the training discussion, eval data near the eval discussion)
+ ## Reproducibility  (absorbs Parameters)
- require Details / "Methodology corrections" / "Why confidence is where it is"
+ drop those; Next steps only when Thomas requests them
+ standalone-framing rule: no "compared to #N" outside Motivation; ban "byte-identical"
```
Target: .claude/agents/upload-verifier.md (+ a paired check in scripts/verify_task_body.py) — what: verify that eval-result files a body's Reproducibility section claims are committed actually exist in that git tree / on the HF data repo before emitting PASS; "local"/on-pod-only is a FAIL. Why: #397's Step-8 upload-verification PASSed (2026-05-29) while the 72-cell derived JSONs lived only on the pod; the pod hit disk-full cleanup and the data was deleted, so the published clean-result is irreproducible and its Reproducibility section falsely claims the 72 files are "committed to main at eb7bc30e" (that commit contains 0 of them). Verbatim (session 1a9086cd): "that Step-8 PASS was effectively false" / "#397 must NOT be promoted as-is." Data-integrity-warning marker already posted on #397. Proposed edit:
```
## Verification checklist
+ - Eval JSONs / figures the body's Reproducibility section claims are committed
+   MUST be verified present (`git cat-file -e <sha>:<path>`) OR on the HF data
+   repo. On-pod-only is a FAIL — the pod is ephemeral. Cross-check every
+   checkable Reproducibility claim (named files exist at the named SHA;
+   pod-terminated marker matches the live API) before PASS.
```
Target: .claude/workflow.yaml (+ scripts/task.py, scripts/verify_task_body.py) — what: the binding-concerns protocol (CONCERNS verdicts become fix-or-explicit-defer, concerns.jsonl persistence, gate concern_deferral_request) is fully drafted + reviewed but stranded on branch worktree-agent-a2da4775a3a1f5fbc, unmerged to main. MERGE decision — review then merge/cherry-pick; not a fresh edit. Why: Thomas asked directly (session 42602fea): "How can we change the workflow so that if a concern is surfaced it must actually be fixed?" after #399's probe-position concern was flagged by code-review (Major #1), deferred as "advisory," and then proved claim-relevant — costing a full re-run. The fix exists (7 commits incl. aa15b2bd3 workflow.yaml: binding-concerns contract) but git log main shows none merged; until merged, CONCERNS still auto-passes and the incident class recurs. Proposed action: review workflow.yaml § concerns_protocol + the 5c/9a decision tables on the branch, then merge worktree-agent-a2da4775a3a1f5fbc to main (or cherry-pick 2880d3feb..710aa5600).

Target: CLAUDE.md (living-docs rule, ~line 14) — what: extend "state facts, not sources" from docs/ to task/issue bodies, and ban spatial/anatomical metaphors in living docs. (Merges the unapplied daily-2026-05-29 proposal #1 with a fresh 05-29 finding.) Why: Thomas, verbatim (session f9b47ab7): "Stop mentioning Dan in notes. Save to CLAUDE.md to just add factual information not the sources" and "#444 and #445 task bodies still say 'Dan (2026-05-29)'... Want those scrubbed too"; plus (session 156d1ff3) "Stop using weird metaphors like spine and hub + spokes. Make this more academic in flavor." The current rule scopes only to docs/open_questions.md. Proposed edit:

- - **Living research docs state facts, not sources.** In `docs/open_questions.md` etc., write the claim directly — no person/meeting attribution. Provenance is the `#issue` evidence trailers; meeting provenance lives in `docs/mentor_updates/`.
+ - **State facts, not sources — everywhere except `docs/mentor_updates/`.** Living docs (`docs/open_questions.md`, `docs/research_ideas.md`, `SUMMARY.md`) AND task/issue bodies write the claim/idea directly — never "Dan said", "per the 2026-05-29 meeting", or any person/meeting attribution. Provenance lives only in the `#issue` evidence trailers and in `docs/mentor_updates/`. Keep the register plainly academic: no spatial/anatomical metaphors ("spine", "backbone", "hub-and-spoke", "scaffold") — name the mechanism directly.

Target: .claude/skills/issue/SKILL.md (cost-confirmation point before Step 6 pod-provision) — what: the post-code-review GPU-spend confirmation must NOT pre-provision the pod; provision only after the user replies approve. Why: #406's /issue session posted the cost note, provisioned a 2×H100 "to save time," then exited awaiting approval and was never re-invoked — orphan pod idle ~24h at ~$5-6/hr (subagent diagnosis, session b07b02ab). The stale-pod audit can't reap a pod the workflow provisioned speculatively before approval. Proposed edit:
```
+ **Cost confirmation does NOT pre-provision.** Do NOT call `pod.py provision`
+ until the user replies `approve`. If the session must exit at this gate,
+ post `epm:awaiting-spend-approval v1` and ensure NO pod exists yet
+ (incident #406: orphan 2xH100 idle ~24h). Alternatively fold this
+ confirmation into the plan-approval gate (id=4) so there is no second pause.
```
Target: .claude/agents/codex-interpretation-critic.md (+ twins codex-clean-result-critic.md, codex-code-reviewer.md) — what: when the sandbox blocks file reads / DNS / HF fetches, the Codex critic must mark the affected lens BLOCKED and NOT emit PASS — today it falls back to the body's own prose and drifts to PASS, which the reconciler then has to clean up. Why: session 0880757a (#381 r1): "Codex interpretation-critic drifts to PASS when sandbox blocks data"; reconciler memory feedback_codex_passes_when_sandbox_blocks_data.md documents the #381 incident. The fix lives only in the reconciler tiebreak today, not in the critic's own prompt. Proposed edit:
```
+ **If you CANNOT read a data file (sandbox read-only / DNS / denied Read or
+ Bash):** do NOT fall back to the body's prose to score that lens. Mark it
+ `BLOCKED — could not read <path>` and do NOT emit overall PASS. If the
+ overclaims lens or the raw-text lens is BLOCKED, the verdict must be REVISE
+ with a `data-access-blocked` note so the reconciler knows PASS was unreachable.
```
Target: .claude/agents/experiment-implementer.md (+ .claude/agents/implementer.md) — what: when posting a 2nd/3rd review-round marker, first read events.jsonl for the highest existing version of that marker key and pass --version <max+1>; post-marker defaults to --version 1 and does not auto-increment. Why: #389 compaction (session 54d1d992): "task.py post-marker doesn't auto-increment version: implementer's round-2 marker was posted as version: 1 duplicating round-1... User feedback: this trap should be in workflow." Confirmed live: task_workflow.py:617 post_marker(version: int = 1). (Robust alternative = auto-increment in the script — see Other problems & notes.) Proposed edit:
```
+ ## Posting review-round markers
+ Before posting a second/third round marker, read events.jsonl for the
+ highest existing `version` of that marker key and pass `--version <max+1>`.
+ post-marker defaults to 1 and does NOT auto-increment; a duplicate version
+ silently breaks round detection (incident: #389 round-2 posted as version 1).
```
Target: .claude/agents/planner.md (+ cross-ref in .claude/skills/adversarial-planner/SKILL.md) — what: the plan should name the hero figure(s) the headline needs AND an "exploratory plot dump" the analyzer over-produces at the end; raise ONE AskUserQuestion at plan time only when the hero view is ambiguous. Why: recurring — Thomas (session d80d1ee7): "What the final result plots is often really bad. I'm thinking that during planning the agent should ask me exactly what to plot. Or maybe it should just plot a lot of things at the end?" #390's TL;DR took ~12 figure-rework rounds. what to plot is absent from planner.md + adversarial-planner SKILL.md. Proposed edit:
```
+ **Figures to produce (over-produce, ask only when ambiguous).** List the
+ specific hero figure(s) the headline needs AND a short exploratory set the
+ analyzer generates at the end (per-cell bars, per-seed scatter, trajectory
+ lines, raw-alongside-residualized). When the hero view is non-obvious, raise
+ ONE AskUserQuestion at plan time. 
```
Target: .claude/agents/consistency-checker.md (comparability table) — what: add a "matched training budget" comparability check. (Carried forward — first proposed in daily-2026-05-29 #2, not yet applied.) Why: session f1d008c0: "Wait no there should be at least a comparable amount of gradient updates. Dispatch a subagent to check if this makes sense." A comparison was matched on epochs/example-counts but not total gradient updates — the confound that actually matters. (Recurred this week in #404's recipe-vs-pair split.) Proposed edit:
```
+ | **Matched training budget** | WARN | When comparing recipes/conditions/cells, total gradient updates (steps x effective batch) should be comparable, not just epochs or example counts. Flag if one condition gets materially more updates than another and ask the planner to justify or rebalance. |
```
Target: .claude/agents/critic.md (Methodology lens) — what: add a marker-dynamics-logging design check. (Carried forward — first proposed in daily-2026-05-29 #3, not yet applied.) Why: session 9e74ceaf: "We don't have information about the dynamics of marker emission right, the fact it's fully saturated at the end doesn't help us too much." CLAUDE.md already requires logging the trajectory, but designs keep shipping that measure only the endpoint. Proposed edit:
```
+ N. **Marker-dynamics logging (marker-implant experiments).** If the experiment implants/tracks a marker, the design MUST log per-step marker log-prob + emission rate as a WandB trajectory. REVISE if the plan captures only end-of-training state — endpoint saturation alone can't distinguish recipes or locate when leakage emerges.
```
Target: .claude/agents/experimenter.md (fresh-pod launch checklist) — what: default save_strategy=steps (with +save_steps/+save_total_limit Hydra +-prefix) for any run estimated >~2h/seed, and mkdir -p logs before redirecting. Why: #382 (session dc47c348): pod died mid-run after ~ $70 with no checkpoints enabled → seeds recovered by hand and Phase-2-relaunched (~$ 215 redo); the fresh-pod relaunch then hit two avoidable startup failures in a row — logs/ didn't exist (redirect failed) and Hydra rejected bare save_steps (needs +). Same class as the already-documented +gpu_id gotcha. Proposed edit:
```
+ Before launching a multi-seed run estimated >~2h/seed: (a) `mkdir -p logs`
+ on the pod; (b) enable checkpointing with `+save_steps=N +save_total_limit=K`
+ (the `+` prefix is required, same as `+gpu_id`) so a mid-run pod death is
+ recoverable. A long run with no save_strategy is a money-loss hazard (#382: ~$215 redo).
```

Target: CLAUDE.md (§ Output format, ~line 252) — what: correct the false claim that the dashboard renders tasks/<N>/artifacts/<slug>.html at a browsable URL — it renders only the task body.md. Why: Thomas was handed .../tasks/402/artifacts/open_questions_v2.md and replied "404 this page could not be found" (session 43934e05); the assistant conceded the CLAUDE.md note was wrong. The stale note still sits at line 252 and will mislead again (same 404 class recurred on the docs route 05-29). Proposed edit:

- reference from events.jsonl's `artifacts` array; the dashboard renders it at `https://eps.superkaiba.com/tasks/<N>/artifacts/<slug>.html`. Pair with the `frontend-design` plugin.
+ reference from events.jsonl's `artifacts` array. NOTE: the dashboard renders ONLY the task `body.md`, not files under `artifacts/`. To give Thomas a browser-viewable doc, put the content in the task body (snapshot the original first) or use a pushed GitHub blob link. Pair HTML artifacts with `frontend-design` for local viewing.

Target: .claude/agents/research-pm.md — what: promote the no-<options>-XML rule from the buried Voice section to a top-level hard output rule. Why: the guard exists (added 2026-05-10) yet the /pm session emitted a 5-option <options>/<option> block 16 days later (session 0081a8a7), which Happy renders as broken cut-off pills (forbidden per feedback_no_options_block.md). Proposed edit:
```
+ ## Output hard rule (read before every reply)
+ NEVER emit `<options>`/`<option>` XML in chat output — Happy renders each as a
+ separate cut-off pill. Present ranked next actions as a plain numbered markdown
+ list. Non-negotiable, every turn.
```
Target: .claude/skills/daily/SKILL.md — what: a manual /daily run must write a complete file including the ## Proposed workflow improvements section, since the 23:27 PT cron refuses to overwrite an existing file. Why: a manual /daily earlier in the day makes the cron no-op, so the file ships missing its proposals section — exactly what happened to the 2026-05-27 daily (older format, no sweep). (Session 5b38d51d.) Proposed edit:
```
+ **Manual runs write the FULL file.** When `/daily` is invoked manually (not
+ via cron), always produce every section including `## Proposed workflow
+ improvements` — the nightly cron refuses to overwrite, so a partial manual
+ file permanently loses the sweep for that day.
```

Living-docs drift

scripts/living_docs.py check → FAIL, 168 problems (exit 1). Categories:

6 appN questions (app1–app6) have an anchor but no parseable > **State:** / **Evidence:** trailer — these question stubs are structurally incomplete.
~95 task #N relates_to '<anchor>' but no question with that anchor exists — task frontmatter still points to the old anchor scheme (a1, b1, c5, e1, h2, app-trigger-discovery, …) that was renamed to the current descriptive slugs (leak-predictor, fact-teach-persona-transfer, …).
~60 bidirectional mismatches — a question lists evidence #N but task #N's relates_to doesn't name that question (e.g. leak-predictor lists #396/#207/#368/#311/#380; fact-teach-persona-transfer lists #192/#389/#390/#407/#381), and the inverse (task #102 relates_to 'app2' but #102 absent from app2's evidence).
Proposal: re-synthesize docs/open_questions.md (run the living-docs updater/backfill) to reconcile the anchor-rename drift + bidirectional evidence ⇄ relates_to mismatches, and fill the 6 appN State/Evidence trailers — needs your ok; not auto-applied. This is a proposal only; /weekly never runs the re-synthesis.

My thoughts

Highlighted results

#390 — Refusal-style negatives install a persona gate that generalises across 9 of 11 OOD eval framings as a verbatim one-line refusal substitution (MODERATE confidence)
#391 — Could not selectively implant sycophancy on Qwen2.5-7B-Instruct: training lifts the trained source and 23 bystander personas in lockstep, with four probable design-level confounds (LOW confidence)