EPS Dashboard

What happened

Big autonomous-experiment day. ~9 /issue sessions ran in parallel (#489, #496, #498, #501, #502, #503, #504, #505, #506). Three reached awaiting_promotion (clean-result drafted + critic PASS): #498 (custom chat-template role token did NOT gate scenario traits tighter than a system-prompt persona; the role token actively suppressed pushback in its own coding scenario — MODERATE), #496 (warmth training gave sub-threshold sycophancy lifts; no source cleared the +0.10 gate — LOW), #492 (the v4/v6 marker-implant floor was an eval-side artifact, not a training-code regression — MODERATE). None were promoted to completed today.
Morning roadmap fan-out. Created #493–#499 from the week's plan (predict fact leakage from cosine/JS, move to realistic contexts/behaviors), plus #503–#506 (leave-one-out / bubble-vs-barrier contrastive negatives, FWFT-27B-survives-benign-SFT). #444 got an inline re-slice: the bystander's own prior on the fact beats/reverses the geometric (cosine/JS) predictor of fact leakage. Archived #392 and #459.
Two hard external walls hit mid-day. (1) Claude Code access: the Ethan Perez Collaborators Org access was removed today, dropping Thomas to a Claude Max 20x plan; autonomous sessions hit "Not logged in", "hit your limit / resets 8pm", and an OAuth wall, stalling several. (2) RunPod $80/hr account cap: #503 and #505 blocked on INSUFFICIENT_BALANCE / over-spend-limit when too many pods were live at once; Thomas added credits and resumed.
A lot of workflow plumbing landed for autonomous-session survival: "autonomous sessions never offer choices — decide and continue", session-survival backstop (keep in-session cron alive past done; stop pods whose driving session died), conservative pod-safety auto-stop + alert + orphan-state GC, cron wrappers put uv on PATH (crons were silently exit-127/dead), and a workflow-fix merge accepting a bare leading PASS in the upload-verification parser (#465). Default autonomous GPU-h cap discussed and raised to 100.
Currently still running: #501 (multi-turn drift supplement to #489, slow Phase 1c) and #502 (#493 bake-off rerun, all 28 layers × probes — manually orchestrated). Blocked: #503, #504, #505, #506. Planning/plan_pending: #488 (whack-a-mole re-plan), #507.

Proposed workflow improvements

Target: CLAUDE.md — what: add the explicitly-requested rule to generate datasets with Claude Code Haiku subagents instead of the Anthropic API. Why: Thomas (session c2c9e281, 23:09Z): "Can you tell them to just use a Claude Code Haiku subagent to generate the data? Also save to CLAUDE.md to not use Anthropic API any more but use Claude Code Haiku subagents instead." Grepping CLAUDE.md + .claude/rules/*.md shows no such rule exists — the "save to CLAUDE.md" instruction was dropped today. Note the tension with the existing Claude as Judge memory (judging uses the Claude API): the instruction says "not use Anthropic API any more," which is broader than data-gen. Scope below to data generation, and flag the judge-path question for Thomas rather than silently overriding it. Proposed edit: (add a Critical Rules bullet)
```
+ - **Generate training/eval data with Claude Code Haiku subagents, not the Anthropic API.** Any data-generation step (synthetic personas, completions, wrong-answer pools, paraphrases) MUST spawn a Claude Code Haiku subagent (`Agent` tool, `model: haiku`) rather than calling the Anthropic API directly — avoids burning metered API spend. The LLM-judge path (Claude Sonnet judge) is a separate question: confirm with Thomas whether it also moves off the API or stays. (Requested 2026-06-05.)
```
Target: scripts/pod.py (+ scripts/runpod_api.py) — what: pre-provision/pre-resume guard against the RunPod $80/hr account spend cap, failing loud BEFORE a run starts instead of mid-run. **Why:** #503 blocked with *"RunPod account over$ 80/hr spending limit on pod-503 resume"* and #505 with *"INSUFFICIENT_BALANCE: 'Renting this pod would put you over your current spending limit ( $80.00/hr).'"* — both AFTER the experiment was underway. With ~9 pods live in parallel, the account cap was hit collectively; the failure surfaced as a mid-run block, and Thomas had to notice and add credits (`54d1d992`: "I added credits. continue"). A pre-provision check (query current account hourly burn; refuse early with the projected total vs the$ 80 cap) converts a mid-run stall into a clean preflight message, and a soft global concurrent-pod cap would prevent the pile-up. Proposed edit:
```
  def provision(...):
+     # Account-level spend guard: sum hourly cost of currently-RUNNING managed pods + this pod's
+     # intended rate; if it would exceed RUNPOD_ACCOUNT_HOURLY_CAP (default 80.0), fail loud here
+     # with the projected total rather than letting RunPod refuse mid-run (INSUFFICIENT_BALANCE).
+     _assert_under_account_hourly_cap(intended_rate)
      ...
```
confidence: medium
Target: .claude/rules/gotchas.md (or CLAUDE.md Pre-launch protocol) + .claude/agents/planner.md §11 — what: require a config-only model-id load check for any model not previously used in the repo, before committing a multi-GPU pod. Why: #506 (Qwen/Qwen3.5-27B) passed 4 code-review rounds + a cap-3 override, provisioned an 8× H200 pod, then died at launch: "qwen35_27b_model_type_unrecognized_by_transformers ... ALL 3 arms failed." The model id is a load-bearing choice (CLAUDE.md already requires grounding load-bearing hyperparameters), but nothing validated that the installed transformers can even instantiate it before spending the 8× H200 provision. (Context: Thomas spent the morning churning between Qwen3.5-27B / 3.6-27B / 3.6-9B / 3-8B / 3-14B — the model ids are new and unvalidated.) A cheap AutoConfig.from_pretrained(model_id) smoke + a planner note when the model id is new to the repo would have caught this on CPU. Proposed edit: (planner.md §11 grounding + a preflight note)
```
+ - **New model id ⇒ config-load smoke before provisioning.** If the plan's `model` is not already
+   used by an existing config/issue in this repo, the plan MUST record a CPU-side
+   `AutoConfig.from_pretrained(<model_id>)` check (does the installed transformers recognize the
+   model_type?) as a preflight gate. Do not provision a multi-GPU pod on an unvalidated model id. (#506)
```
confidence: medium
Target: .claude/skills/issue/SKILL.md (autonomous section) + scripts/spawn_session.py — what: close the residual path where --auto happy sessions still surface the plan-approval multiple-choice gate. Why: Thomas (83a3ba27, 17:26Z): "The happy coder sessions still asked for plan approval (via multiple choice question) even though this isn't supposed to happen anymore. Please figure out why." Today's commits ("autonomous sessions never offer choices — decide and continue", 02:31) addressed part of this, but it was still firing at 17:26 — either the affected sessions predate the fix or a path still raises AskUserQuestion when EPM_AUTONOMOUS_SESSION=1. Verify every plan-gate exit in autonomous mode routes to auto-approve/park (never a choice menu), and that spawn-issue --auto reliably exports the autonomous env flag. Proposed edit: (audit + assert, sketch)
```
- <plan-approval AskUserQuestion in autonomous mode>
+ if EPM_AUTONOMOUS_SESSION: auto-approve (≤cap) or park at plan_pending — NEVER AskUserQuestion.
+ # add a workflow_lint check: no reachable AskUserQuestion under the autonomous branch.
```
confidence: medium
Target: .claude/agents/experiment-status.md (+ stuck-diagnoser.md) — what: make "is issue N running?" reconcile against the live Happy/Claude session set, not just events.jsonl markers. Why: Thomas (dc35da49, 19:11Z): "When you check the status do you actually check if there are any happy/claude sessions running for the issue or just look at the markers?" and (c76f8b90, 17:44Z): "I want to step in if the session is blocked but we can't trust the session to always properly post blocked when it's blocked." The autonomous-session watcher already flags ALIVE-BUT-STALLED (it fired on #503/#504/#505: "dead bg-Bash chain inside a still-live Claude process"), but the status-report path the assistant uses on demand should cross-check spawn_session.py list (live sessions) against the latest marker so a silently-dead session isn't reported as "running." Proposed edit:
```
+ ## Liveness reconciliation (MANDATORY)
+ For each issue, cross-check the latest events.jsonl marker AGAINST `spawn_session.py list`.
+ Report: marker-status, live-session? (yes/no), and stale-gap minutes. A non-terminal marker with
+ NO live session = "stalled/dead session", not "running".
```
confidence: medium
Target: .claude/skills/issue/SKILL.md Step 6d.2 cron arming (+ .claude/workflow.yaml) — what: verify every armed backstop cron uses /issue-tick <N> (not /issue <N>) and reconsider the 10-min interval against the 5-min cache TTL. Why: Thomas (9893cfe3, 21:25Z): "Didn't we fix this: The /issue backstop cron fires every 10 min; the cache TTL is 5 min. Every idle tick re-writes the entire ~200K+ prefix cold at 1.25× price." and earlier (c76f8b90, 17:29Z): "Is rerunning the issue skill really the most efficient way to do this? Doesn't it bloat the context a lot?" + (fd991dde, 17:47Z): "Do we really have to run issue 489 every 10min? Can't it be a subtler nudge." The issue-tick lightweight skill already exists and SKILL.md says Step 6d.2 arms /issue-tick — so the per-tick payload is small. Residual issue: a 10-min interval against a 5-min TTL means the prefix is always cold. Confirm no in-flight cron still points at the heavy /issue <N>, and consider lengthening the idle interval or making the tick event-driven. Proposed edit:
```
- CronCreate(... prompt="/issue <N>" ... schedule="*/10 * * * *")
+ CronCreate(... prompt="/issue-tick <N>" ... schedule="*/10 * * * *")   # lightweight tick, not the 44K skill
+ # consider: longer idle interval (>5min TTL is moot; pick 15-20min) to cut guaranteed cold-cache misses.
```
confidence: low

My thoughts

Highlighted results

no results promoted today

Daily — 2026-06-05 (heavy autonomous run; Claude-access + RunPod-balance walls; 3 results reached awaiting_promotion, 0 promoted)

What happened

Proposed workflow improvements

Other problems & notes

My thoughts

Highlighted results