Daily — 2026-05-31 (on-policy marker re-run lands rank

What happened

• Clean-result spec migrated to v2. #454 moved the body spec to the 2-content-section nested design (Human TL;DR / TL;DR with ### Motivation + ### What I ran + ### Findings→#### per result / Reproducibility; confidence in the H1 title only, no Details/Methodology-corrections H2). #455 reconciled the binding-concerns protocol branch onto current main via compose. Both auto-completed (ensemble PASS + test-verdict PASS). This was driven by your live redesign of the #432 body (kill the Details section, params→Reproducibility, raw data inline at the section it's discussed, training/eval data as 5-example dropdowns + link to full).
• The #432 off-policy bug got root-caused and fixed end-to-end. #432 measured marker emission with a teacher-forced fixed-stub probe at pos 0 → you promoted it not-useful. #456 re-ran the same SFT on-policy: the trained software_engineer source emits ※ on 0.90 of its own answers, rank #1/28 (parked awaiting_promotion). Two CLAUDE.md rules landed (measurement-validity #432→#456 incident; "measure marker leakage as continuous on-policy log-prob, not binary emission rate").
• Codex code-reviewer gate-hopping fixed (commit 0e006d933) after you flagged it twice stopping at a different procedural gate each round and never reviewing actual code.
• Four clean results parked at awaiting_promotion: #456 (on-policy, above), #407 (fictional arm replicates #389/#390 persona-gating; obscure-real arm marked BUGGED — contaminated by stale CJD paraphrases), #406 (base-model output divergence negatively predicts SFT transfer; N=16 after C2–C5 dropped), #448 (contrastive-negative geometry steers leak localization — you caught a sign-flip in the H3 heading post-park).
• In-flight + recovery: #408 (multi-turn ※ install — 3 seeds trained after 6 sequential bug fixes + ~41h idle bleed ≈ $103, now in eval recovery), #444 (fire-hydrant fact regime, replanned v3→v5 at your direction, Phase-0 preflight deadlock), #451 (C-axis gate fix re-run), #458 (18-dataset EM predictor sweep, recovering at 15/18 seed-0). New proposed follow-ups: #459 / #460 / #461 (continuous-logprob + cosine-JS predictor) and #453 / #457.
• #458 checkpoint-deletion fix in progress on branch fix/sweep-ckpt-persist (trainer.py adapter-persist + test_adapter_persist.py + CLAUDE.md delete-after-eval rule) — currently uncommitted.

Proposed workflow improvements

1. Target: CLAUDE.md (Pods § "Crons.") — what: make a managed pod whose run is DONE but task is parked (verifying / interpreting / reviewing / awaiting_promotion) count as stale, so the daily audit terminates it. Why: three pods bled idle for days because the stale audit only auto-terminates EXITED pods. #382 ran ~3 idle days (pod-382 ... ran ~3 idle days because its task was at awaiting_promotion), #397 idle with disk 100% full, #404 idle ~2d — all RUNNING, not EXITED, so the cron skipped them. A workflow-fix-candidate for scripts/pod_audit.py was emitted on #382 but never applied (no epm:workflow-fix-applied). Real money. Proposed edit:

   - **Crons.** Stale-pod audit 09:37 daily (auto-terminate EXITED >24h; also runs on `pod.py provision`).
   + **Crons.** Stale-pod audit 09:37 daily (auto-terminate EXITED >24h; also runs on `pod.py provision`).
   +   ALSO flags a RUNNING managed (`epm-issue-<N>`) pod as stale when its task status is run-complete-but-not-terminated
   +   (verifying / interpreting / reviewing / awaiting_promotion) for >6h — the run is done, the pod is just bleeding.
   +   Implementation: re-dispatch the pending #382 `workflow-fix-candidate` against `scripts/pod_audit.py` (audit-stale).
   

2. Target: CLAUDE.md ("Push through bugs in recovery mode" §) — what: add an explicit per-run failure-class budget so a single experiment can't burn dozens of GPU-hours across many auto-fix+relaunch cycles before escalating. Why: #408 hit BUG #6 and 41h idle ($103) before the CONTINUE escalation; the existing "≥3 fundamentally different fixes failed" line is about blocking strategies, not about a long-running run racking up relaunches. The counter isn't enforced per-run. Proposed edit:

   - Escalate only when (a) the fix changes experiment scope, (b) the fix is irreversible/high-cost ... (c) ≥3 fundamentally different fixes failed, or (d) the bug points to a real factual question only Thomas can answer.
   + Escalate only when (a) the fix changes experiment scope, (b) the fix is irreversible/high-cost ... (c) ≥3 fundamentally different fixes failed, or (d) the bug points to a real factual question only Thomas can answer.
   +   Hard per-run cap: after **3 distinct `failure_class` root-causes on the SAME run** (count the `epm:failure` markers), STOP auto-fix+relaunch and escalate with the running idle-cost so far — do not keep patching a run that has failed for 3 different reasons.
   

3. Target: .claude/agents/clean-result-critic.md (or scripts/verify_task_body.py) — what: a specific artifact COUNT in the Reproducibility section must be verified against what was actually produced, or stated qualitatively. Why: #397's clean-result body carried false reproducibility claims (Body claims '72 metrics.json + 72 logprob...') and was flagged DO-NOT-PROMOTE only by a human spot-check at termination time — both critics had already PASSed it. Proposed edit:

   + ### Lens 14b — Reproducibility artifact counts must be real
   + Any concrete artifact count in `## Reproducibility` (e.g. "72 metrics.json", "240 raw completions")
   + must trace to a verified file listing (HF repo / git path) cited in the same section. If the analyzer
   + cannot verify the count, it must be phrased qualitatively ("per-cell metrics.json + logprob JSON") — never
   + a fabricated number. FAIL on an unverifiable specific count.
   

4. Target: .claude/agents/interpretation-critic.md (Lens 6 — plot-prose match) — what: explicitly check the SIGN/DIRECTION of a claimed relationship against the figure, not just that a figure exists and matches the metric. Why: #448's H3 heading read "Bystanders FARTHER from the trained negatives leak MORE" — backwards relative to the figure and the prediction; you caught it post-promotion after both interp-critic and clean-result-critic PASSed (SIGN-FIX (user-caught, post-promotion-park)). Proposed edit:

   + Lens 6 sign-check: for every "X predicts/increases/reduces Y" or "CLOSER/FARTHER → MORE/LESS" claim in a
   + heading or read paragraph, confirm the SIGN matches the plotted slope/ordering. A correct magnitude with a
   + flipped direction is a REVISE, not a pass. State the figure's actual direction in the verdict.
   

5. Target: .claude/agents/code-reviewer.md (or .claude/agents/experiment-implementer.md) — what: flag when the implementation measures a different quantity than the plan's declared dependent variable (e.g. teacher-forced fixed-context probe vs the planned on-policy generation). Why: #432's eval silently "swapped in a fixed stub" answer and scored an off-policy probe instead of the on-policy emission the plan intended — you asked (16:00) "how can we prevent this kind of design decision in general?". Today's measurement-validity CLAUDE.md rules govern the plan; this closes the implementation-drift side (an agent quietly substituting a cheaper proxy at code time). Proposed edit:

   + ## Step 0.8 — DV-fidelity check
   + Compare what the eval code actually computes against the plan's declared dependent variable + measurement
   + (construct / metric / on-distribution, per CLAUDE.md measurement-validity). If the code measures a cheaper
   + proxy than the plan declared (teacher-forced vs on-policy, fixed-context vs generated, single-position vs
   + trajectory) and the plan did not opt into it, that is a substantive FAIL — name the substitution.
   

Other problems & notes

• #408 — six sequential experiment-code bugs (source_corpora_too_verbose_for_fixed_seqlen_multiturn_prefix, eval_corpus_floor_too_strict, sentinel-filter misses) plus a transient ~10-min Sonnet preflight hang with GPUs idle. Experiment code, not workflow surface. Suggested action: the multi-turn data-gen design (fixed seqlen vs verbose source corpora) needs a rethink before the next multi-turn run — file a design follow-up rather than patching cell-by-cell. Already escalated to you (CONTINUE decision); P5 above is the workflow guardrail.
• #452 + #458 both hit bitsandbytes missing from the issue-branch uv.lock (the turner_em recipe uses optim=adamw_8bit). Recurred twice the same day. Suggested action: preflight should assert the chosen recipe's optimizer deps are importable before any subprocess spawns (fix via experiment-implementer / preflight.py).
• #444 Phase-0 preflight deterministic deadlock in _smoke_check_train_lora_config (reproduced 2×, GPUs idle). Suggested action: fix the import/deadlock via experiment-implementer; it's a reproducible hang, not flakiness.
• #451 round-1 crash — TrainLoraConfig.__init__() got an unexpected keyword argument lora_target... across all 72 cells (config/API drift). Suggested action: fix via experiment-implementer.
• #456 partial-port crash at smoke_train (format_dataset list-shape port from #432 recipe) — fixed at implementation round 3. Note only.
• 2 malformed tool calls (session dc93f209, 15:42 & 15:51 — "Your tool call was malformed and could not be parsed"). Transient model errors; no action.
• Dashboard — two summary documents on the eps.superkaiba.com overview were confusing (session 73072529). You chose option B (auto-render the overview from the summary + open/closed questions + results) and asked to plan it comprehensively. Dashboard work, separate track (not workflow surface).
• #382 workflow-fix-candidate for scripts/pod_audit.py was emitted but never applied — folded into P1; re-dispatch workflow-improver.
• #458 adapter-persist fix is uncommitted on fix/sweep-ckpt-persist (trainer.py, test_adapter_persist.py, CLAUDE.md). Suggested action: finish + commit, and run the WandB recovery check on #458's already-deleted checkpoints that you asked for (23:23).

My thoughts

Highlighted results

• #432 — Expanding contrastive negatives from 2 to 9 lifts the trained source persona from rank 26 to rank 7 on the marker leaderboard, by suppressing the eight in-panel trained negatives rather than boosting the source (MODERATE confidence) — promoted not-useful (off-policy probe; superseded by #456).
• (awaiting promotion, today's real science): #456 — the trained persona writes the marker on 90% of its own answers, rank #1/28 (MODERATE); #406 — base-model output divergence negatively predicts SFT marker transfer (MODERATE); #448 — widening contrastive negatives reduces bystander leakage (MODERATE); #407 — fictional arm replicates persona-gating, obscure-real arm BUGGED (LOW).