Daily — 2026-06-03 (10 experiments parked at awaiting_promotion, 6 workflow fixes auto-shipped, 0 promoted)

What happened

• High experiment throughput. Ten experiment tasks reached awaiting_promotion today: #405 (marker leakage vs K source personas), #444 (invented-fact localization, 4th fact regime + 5-way re-judge), #464 (persona-in-role-header marker localization), #466 (slice-aware JS/cosine leakage predictor), #468 (full n=18 cosine→EM re-run), #470 (consolidated with #411: marker setup can no longer predict sycophancy transfer), #474 (restoring broad contrastive negatives recovers the divergence→transfer correlation), #475 (CoT-scaffold marker install on Qwen3.5-27B — clean negative), #478 (training the marker into K=1→8 sources raises leakage but doesn't flatten the distance gradient), #480 (token marker doesn't predict sycophancy leakage). Nothing was promoted to completed — the 70-deep awaiting_promotion backlog grew.
• Active runs at day's end: #471 (in-context-spec with contrastive negatives — relaunched on a fresh pod after a crash + supply-constrained resume), #477 (contrastive-negative count vs implant-strength — pivoted from a failed LR-decoupling lever to a step/epoch lever, v4 in code review), #459 (EM-dataset generalization — autonomous loop stalled, see below).
• 6 workflow fixes auto-shipped to main via the workflow-fix-on-bug protocol: pod.py terminate upload-verification guard (#444, 4f5d34bf4), failure_classifier.py now reads torch's wrapped "Original Traceback" so collator/dataset errors route code not infra (#480, 180486709), pod.py terminate selects pods via the live API + fails loud on survivors — closes the stale-pod-id money leak (#475, 21f3e4b79), SKILL.md --step 9 doc gap (#474), SKILL.md Step 10d auto-merge guard (#479, b5c5a96d), poll_pipeline.py fix (#405).
• Two feature changes landed on main: (1) issue worktrees now auto-merge to main at the terminal state (Step 9b / Step 10d, no prompt) — retired the worktree_merge inline gate, dropping inline gates 8→7 (65373a896); (2) session-topology work — /issue N auto-titles its Happy session, spawn_session.py list enrichment, and a PM --auto mode that spawns autonomous per-issue sessions auto-approving sub-24-GPU-hour plans (820a1f5b3, a0ba005e4, 3f91d4d55).
• MarkerOnlyDataCollator default flipped tail_tokens=32→0 after a saturation deep-dive concluded the loss mask (not training dose) was the load-bearing lever behind apparent marker saturation.

Proposed workflow improvements

1. Target: .claude/skills/issue/SKILL.md + .claude/agents/upload-verifier.md — what: stop issuing set-status <N> uploading; uploading is not a task.py enum value, so every such call is rejected. Why: Bit two sessions today. #466: "uploading isn't a valid status enum ... the correct enum for the upload-verification step is verifying" (session 53909073). #475: "the skill's 'uploading' label rolls under running/verifying" (session 67142484). Verified: the canonical enum is proposed planning plan_pending approved running verifying interpreting reviewing awaiting_promotion completed blocked archived — no uploading. The imperative command sites are SKILL.md:1699, 2030, 2035, 2115 and upload-verifier.md:6, 280. (The descriptive flow-diagram uses at SKILL.md:115/219/265/3014 already annotate uploading (verifying) and can stay.) Proposed edit:

   # SKILL.md:2030
   - 3. If `epm:results` exists, move status to `uploading` and proceed to Step 8.
   + 3. If `epm:results` exists, move status to `verifying` (the upload sub-phase;
   +    "uploading" is NOT a task.py enum) and proceed to Step 8.
   # SKILL.md:2035 / 2115 / 1699: replace each imperative `status:uploading` with `status:verifying`
   # upload-verifier.md:6
   - advancement from status:uploading to status:interpreting
   + advancement from status:verifying to status:interpreting
   # upload-verifier.md:280
   - Stay at `status:uploading`. List the remediation commands.
   + Stay at `status:verifying`. List the remediation commands.
   

2. Target: .claude/rules/marker-leakage-measurement.md (+ mirror the token-id qualifier in CLAUDE.md:18) — what: the marker-leakage measurement rule is missing three footguns that each crashed or contaminated an experiment today. Fix all three in one pass (same file, same "measurement recipe" theme). Why:

   • (a) Hardcoded token id 83399 is Qwen-2.5-7B-specific. #475 trained the marker on Qwen3.5-27B where ※=80522 (bare ※=61531); the rule's assert ... == [83399] would fail on any non-7B model and the orchestrator had to discover 80522 mid-run (session 81ce8532).
   • (b) The count==1 "marker appears once" invariant breaks on a TRAINED model's own R. #472 forked it verbatim from #448 (where R is marker-free base output) and crashed twice: "the trained model's own on-policy response collapsed to ~256 repeated ※ tokens ... the count==1 invariant ... breaks for trained-model R that contains markers" (session b77d199a). Also needs a repetition-collapse guard (exclude degenerate max-leakage probes from the graded regression).
   • (c) tail_tokens=0 is unnamed. The saturation deep-dive (session 67921680) landed on the loss mask: "We always want only loss on last token" — tail_tokens>0 silently puts loss on response tokens and contaminates the on-policy DV. The rule documents the #432→#456 incident but never names MarkerOnlyDataCollator(tail_tokens=0). Proposed edit: (sketch — workflow-improver refines line anchors)

   # §"Default marker token" (~line 21-26)
   - assert `tokenizer.encode(MARKER_TEXT, add_special_tokens=False) == [83399]` before any subprocess spawns
   + The id 83399 is Qwen-2.5-7B-SPECIFIC (Qwen3.5-27B: ` ※`=80522, bare `※`=61531).
   + Assert the marker is a SINGLE token under the experiment's ACTUAL base-model
   + tokenizer (`len(encode(MARKER_TEXT, add_special_tokens=False)) == 1`), record that
   + per-model id, and use it downstream — never hardcode 83399 for a non-7B model.
   # §"Measurement recipe" step 2 (~line 46)
   + Mask loss to ONLY the marker token via `MarkerOnlyDataCollator(tail_tokens=0)` (canonical default).
   + NEVER use tail_tokens>0 for a leakage DV — it trains the response, contaminating the on-policy measurement.
   # new subsection after the recipe block, before "## #432 → #456 incident"
   + ## Measuring leakage on the TRAINED model's own R
   + - Do NOT assert the marker appears exactly once — the count==1 invariant is valid
   +   only for marker-FREE base-model R (the #448 regime); forked onto trained-model R
   +   it crashes (count=257). Locate the post-response slot without assuming cardinality.
   + - Guard against marker-repetition collapse: once the marker saturates the argmax,
   +   greedy gen can loop to max_new_tokens (#472: ~256 repeated ※). Flag r_collapsed
   +   probes as degenerate max-leakage and EXCLUDE from the graded regression.
   

3. Target: .claude/rules/gotchas.md — what: add a gotcha for vLLM monkey-patching transformers config classes in-process (distinct from the already-documented GPU-memory teardown gotcha). Why: #475 burned ~5 wrong fixes chasing it: after an in-process LLM(...), a later AutoModelForCausalLM.from_pretrained crashes with AttributeError: 'Qwen3_5Config' object has no attribute 'vocab_size' (hit 5×). "vLLM in-process patches transformers; fix = vLLM-free subprocess logprob worker (a8b789867). Earlier wrong fixes (config guard, drop-config) were chasing the load." (session 81ce8532). Proposed edit:

   + - **vLLM also monkey-patches transformers CONFIG classes in-process at engine init.**
   +   After an in-process `LLM(...)`, a later `AutoModelForCausalLM.from_pretrained` can
   +   crash `AttributeError: '<Model>Config' object has no attribute 'vocab_size'`
   +   (Qwen3.5-27B, #475). NOT a config-construction bug in our code — do NOT chase it
   +   with config guards / drop-config / attn_implementation tweaks (#475 burned ~5 wrong
   +   fixes). Fix: run the HF/logprob path in a vLLM-FREE subprocess and IPC the result.
   

4. Target: .claude/skills/issue/SKILL.md (Step 6d.2, backstop-cron auto-arm) — what: require the resuming orchestrator to re-arm the backstop cron when it finds an epm:run-launched marker with no matching CronList entry. NOTE: overlaps the durable-session work Thomas routed to /plan today; if that plan supersedes this, drop it. Why: Hit twice. #478: run finished cleanly at 22:14Z but "no backstop cron armed ... the poll chain died" and the pipeline sat idle for hours (session c393226f). #471: on recovery CronList returned "No scheduled jobs"; the pod ran/idled ~10.5h (session 403e54d7). The auto-arm is the safety net for exactly this, but a launch-then-die race leaves no cron. Proposed edit:

   + **Backstop re-arm on resume.** When re-invoked on a task already at `running` with an
   + `epm:run-launched` marker, ALWAYS `CronList` and confirm a backstop cron exists for this
   + issue. If the marker is present but no cron is (the launching session died in the
   + arm-vs-dispatch window), RE-ARM the 10-min backstop before polling — otherwise a
   + cleanly-finished run can sit un-advanced indefinitely (#478: finished 22:14Z, idle hours).
   

5. Target: .claude/skills/issue/SKILL.md (park-marker / Step 9b) — what: pin the exact valid step-id token for the awaiting_promotion park marker; bare 9 / 9b are rejected by post-marker. Why: Recurring low-grade friction — the orchestrator guesses an invalid step id, gets rejected, re-posts. #468: "the step marker rejected 9 — re-posting with the valid step id"; #466: "step id should be 9a-bis. Re-posting the park marker correctly." (sessions c7f3d370, 53909073). Proposed edit:

   + When posting the awaiting_promotion park marker, use step id `9a-bis` (the final review
   + gate) — NOT bare `9` or `9b`; integer / non-canonical step ids are rejected by post-marker.
   

6. Target: .claude/workflow.yaml + .claude/agents/code-reviewer.md — what: reconcile the raise-concern --summary cap: docs say <80c> but the code cap is 200, so reviewers guess wrong. Why: #470 reviewer hit ValueError: summary too long (231 chars; max 200) (session ae9d3712). code-reviewer.md:161 and workflow.yaml:872 both say <80c>; the real task_workflow.py cap is 200. Either tighten the code cap to the documented 80 or update the docs to 200 — right now docs, code, and reality (231) all disagree. Proposed edit: update the two doc sites from <80c> to <=200c (docs previously said 80; real cap is 200), or (preferred) keep 80c as the soft guidance and add (hard cap 200).

7. Target: .claude/rules/code-style.md — what: note that load_dotenv() must be called with an explicit path in heredoc/stdin one-liners. Why: Crashed twice today — uv run python - <<'PY' ... load_dotenv() fails in find_dotenv() (no __file__/stack frame to walk from stdin); each time self-fixed with the explicit path (sessions 935a6bfc, ae9d3712). Proposed edit:

   + In heredoc / `python -` stdin one-liners, call `load_dotenv("/home/.../explore-persona-space/.env")`
   + with the explicit path — bare `load_dotenv()` crashes in `find_dotenv()` (no __file__ to walk from stdin).
   

Other problems & notes

Reliability / cost (highest):

• Per-issue sessions don't auto-continue when the loop dies — the day's #1 reliability item. #471 and #480 both died mid-run (429 killing the polling tick with no re-arm; the durable=False backstop cron dying with the session process; respawned Happy sessions coming up as idle shells that never re-attach /issue <N>). Thomas: "I want them to never die and continue automatically. Devise a plan." He routed it to /plan (3 rounds Claude+Codex), still drafting at day's end. Suggested action: resume that plan first thing tomorrow. Do NOT pre-empt with a quick diff (proposal 4 is the narrow mitigation; the plan is the real fix).
• #475: dead session left a 4×H100 pod billing ~6.5h. A prior /issue 475 session died during eval; the sweep had actually finished cleanly but never posted epm:results, so the pod ran idle. Compounded by pod.py terminate --issue 475 killing a stale local pod_id (ghost) while the real pod survived. Suggested action: covered by the landed pod.py terminate live-API fix (21f3e4b79); the root cause is the same dead-session-no-results pattern as #480 (both trace to hand-driven sweep drivers that write no standard sentinel).
• #480: unmonitored crash idled an H100 ~3.5h. Crashed on a max_length=1024 collator-truncation bug; the poll chain died and no backstop cron was armed. Recovered (fixed max_length→2560 + build-time <|im_end|>-survival guard, re-ran, armed cron). Suggested action: same root as the auto-continuation item; the failure_classifier fix (180486709) at least routes it code now.
• #459: autonomous /issue 459 loop stalled on its first turn with 529 Overloaded and produced zero work (session 5f393ecd, PM-spawned via 679b7c4e). The new PM --auto autonomous-session feature has no retry/backoff on a startup 529/429. Suggested action: confirm the /loop autonomous session treats startup 529/429 as retryable (re-fires next tick) rather than dying on it.
• #467: hit the RunPod $80/hr INSUFFICIENT_BALANCE cap — 7 concurrent pods (incl. #472's 16×H100 and a stale EXITED pod-465 idle ~15h) consumed the account hourly spend cap; a legitimate AskUserQuestion halt (raising the cap needs Thomas's account). Suggested action: the EXITED pod-465 sitting 15h suggests the 09:37 stale-pod audit hadn't caught it (likely <24h); consider catching budget-eating EXITED pods faster.
• Two /issue 471 orchestrations ran in parallel (session 78a6554b) — a second session independently took over #471, churning pods and posting markers the first didn't write. The topology rule says one session per issue but nothing enforces it. Suggested action: a lightweight per-issue lock or launch-time live-competing-session check (architectural — touches session spawn; noting, not diffing).

Infra flakiness (mostly recovered):

• RunPod SUPPLY_CONSTRAINT resume failures ×4+ (#467, #471, #475, #480) — pod.py resume never relocates, so a stopped pod whose former host has no free GPU is effectively dead; each recovered via terminate + fresh provision (sometimes losing local-only prep, e.g. #480's Phase-0 R-gen). Suggested action: upload expensive prep artifacts to HF so a fresh pod can recover them.
• Stale pods.conf clobbered by a concurrent session → false-dead pod (#468) — the poller reported the pod dead because a concurrent write reverted pod-468's host to a terminated pod's IP; the run was healthy. Mitigated by per-tick live-API host re-assertion (concurrency race, not a single-file doc fix).
• #475 27B pod-infra chain (session 81ce8532) — MooseFS errno 116 stale-file-handle during uv sync, /dev/shm noexec (venv .so imports failed → moved venv to /opt), torchvision ABI mismatch (torchvision::nms does not exist → reinstall +cu129), Phase-1 OOM (two procs co-resident on one H100). All worked through under "keep going until fixed." Suggested action: if 27B becomes routine, bake a "27B pod intent" preset (driver/CUDA/venv-on-/opt).
• #478 / #474 pod SSH refused 5+ min post-provision despite API=RUNNING — sshd came up then died or endpoint drifted; resolved via pod.py config --sync + re-probe. Same class as the known "verify hostname after resume" gotcha.
• #467 control-character JSONL corruption during smoke — NULL/\x01-\x08 bytes surviving str.strip() in a generated JSONL (likely a MooseFS partial write; preflight logged a Bad file descriptor EDQUOT probe warning). Fixed with a fail-loud _loads_jsonl_line helper (bfaa0bd05). Suggested action: consider a regen-on-corrupt guard in the data-gen entrypoint.
• HF_TOKEN not loaded in non-interactive pod shell; rsync absent on pods; Bash SSH to RunPod proxy timed out (#475, #467) — uv run python one-offs skip setup_env() so HF Hub calls fail without source .env; fell back to tar-over-ssh and MCP SSH. Recurring pod-env quirks.

Experiment-code bugs (resolved this session):

• #472 train_cell.py:131 hardcoded gpu_id=0 → train/sft.py set CUDA_VISIBLE_DEVICES from it → all 4 concurrent cells piled onto GPU 0 → OOM. The +gpu_id=N gotcha is in CLAUDE.md but the experiment-implementer didn't thread it into the new launcher. Fixed round-3 (7b5405443).
• #472 vLLM init OOM on the train→eval same-GPU handoff — fixed inline (gpu_memory_utilization=0.60, 6758bb4c).
• #479 smoke caught a c472_anchor dispatcher bug — the --smoke path scheduled a cell name inherited from the #472 fork that i479_run_cell.py doesn't know; smoke gate aborted correctly before the full run. Fixed round-5.
• #478 design validator over-strict — required all 55 personas (incl. 2 training-only conversational-default negatives) in the distance matrix; fixed inline.
• #471 MarkerOnlyDataCollator(suppress_at_post_response_slot=True) ValueError on cond2_k3 — the k=3 in-context-demo negative row had 444 loss-bearing positions, none <|im_end|>; fail-fast assertion fired correctly, bounced to experiment-implementer.
• #444 fact-builder crashed 4× on over-aggressive token-isolation guards (Elk County Courthouse fact) — fixed iteratively; if a 5th fact-regime reuses this builder, narrow the guard to value tokens up front.

Codex ensemble transient races (all self-recovered per spec):

• Codex twin dispatch exit code 4 / No job found post-spawn probe on #464 (×3) and #467; spawn job-id mismatch on #477 (spawned task-mpxt32gl-..., probed task-mpxt32wr-...). All fell back to Claude-only or retried once. Suggested action: if recurring, add an internal retry-once on the immediate post-spawn No job found before codex_task.py exits non-zero.
• Codex interp-critic stalled / no-show on #466 (force-cancelled, log untouched 604s) — no-show fallback worked.

Process / UX:

• Analyzer hit context exhaustion mid-work on #466 (258k tokens / 40 tool calls), generated 3 figures but never posted epm:interpretation or wrote the body; recovered by re-spawning pointed at the figures. Suggested action: leaner analyzer brief, or checkpoint figures + partial body before the heavy prose pass.
• Dashboard body editor overwrites the whole body.md (#468) — Thomas edited the Human TL;DR in the dashboard and it silently replaced the entire body, dropping the concrete-numbers version ("I think i may have overwritten your changes. Please reapply them"); recovered from a /tmp backup. Dashboard UX (out of the bucket-1 file set) — consider a block-level editor or confirm-on-full-replace.
• #470/#411 consolidation — Thomas asked to combine into one issue headlined "with the new marker setup we can no longer predict sycophancy transfer with cosine similarity or JS divergence," plus next-steps on why. Done.
• #405 needed a substantial human reframe — Thomas rewrote title/body around his TL;DR (two takeaways, one plot each, on-policy log-prob not KL) and had to ask twice to clarify opaque cell-composition notation ("8 single / 6 random pairs / 6 random quads / 1 octet"). Result-specific editorial taste; existing concision + no-opaque-codes rules cover the principle.
• #464 chat jargon friction — Thomas twice said "I don't understand the padded condition" / "I don't understand your natural paired companion." Soft reminder: name conditions in plain English in chat too, not just clean-result bodies.
• src/.../train/sft.py is a serial contention point — #474 and #478 both hit MarkerOnlyDataCollator merge conflicts the same day (branches far behind main). Suggested action: land future collator edits to main early/standalone rather than carrying them on long-lived issue branches.
• CPU-only Phase C regression ran on the H100 pod (#468), idling the GPU ~1h. Suggested action: move bootstrap/permutation regressions to a local/CPU pod.
• Constellation Slack MCP tokens expired (auth.test → invalid_auth); worked around by reading Dan's comments via Beeper (bridged). Suggested action: Thomas re-extracts the xoxc/xoxd Constellation tokens into .env.
• Happy agent.key missing → happy resume dead-end (session e55e7ce7) — name collision (happy reads ~/.happy/agent.key but never writes it) + a hand-patched happy bundle. Host-tooling, outside the EPS surface; track separately.
• Personal-assistant work (answering Dan's Slack comments via Beeper) run inside ~/explore-persona-space/ (session add73de2) — minor topology slip vs the global "keep PA work out of the EPS dir" rule.
• #482 clarifier cleanly interrupted by Thomas — #482 has real unmet dependencies (#446 unrun, #467 still running); resume once #467 lands. Not an error.

My thoughts

Highlighted results

• no results promoted today