Daily — 2026-06-08 (compute-infra expansion + ~40 in-session workflow fixes; 0 promotions)

What happened

• Compute-infrastructure expansion (the day's headline). Brought three new compute backends into reach: Mila cluster SSH access (public key added, IT ticket filed), Compute Canada / DRAC (alliancecan, Duo MFA + robot key for unattended runs), and a GCP account with **$100k GFS credits** (Emanuel's `2024 GFS Cloud Program`, valid through **2026-08-02**) plus$12k enhanced-support credits. Started a SLURM-routing integration so the autonomous workflow tries free academic compute first and falls back to RunPod/GCP only when those are full. Sent the RunPod cap-increase email, the Mila IT email, and the DRAC access request.
• ~12 experiments active or diagnosed today. Running/approved set: #488 (distance→transfer, non-saturated), #504→#530 (de-saturated bubble-vs-barrier rerun), #506 (FWFT marker on Qwen3.5-27B), #514 (LoRA-vs-FT leakage, r=0.994 per-persona), #517→#528 (non-saturated trait + role-header vs system prompt), #518 (refusal + EM leakage prediction), #519→#520 (superposition / steering-vector framing), #511→#521/#522/#523, #524 (directional/asymmetric predictors on a unified instruction+ICL panel), #527, #531, #532. Several reached awaiting_promotion (#504, #509, #514) but none were promoted to completed, so today has no headline results.
• Methodology hardening. A marker training-recipe deep-dive (the over/under-training LR dial: lr≥1e-4 → unconditional repeater; clean window lr≤5e-6, strength via training STEPS not rank) produced .claude/rules/marker-training-recipe.md + the band-stop callback. A long predictor-zoo / evaluation-metric thread worked through asymmetric predictors (whitened/Mahalanobis cosine, one-way KL over response tokens, full-response JS), Gaussian-KL-as-best-predictor, and rebuilt #524 on the #502 data/recipe with a 100-question eval pool. Dan's activation-vector framing (context vector v_c → behavior vector v_b generalization) was worked through in LaTeX.
• ~40 workflow fixes auto-landed in-session via the workflow-fix-on-bug protocol (now auto-apply by default): KL-vs-logprob marker-DV ban, report-both-logprob-and-logit, data-realism hierarchy, replication-fidelity enforcement, reuse-trained-artifacts fitness check (+ consistency-checker reuse-smuggled-variable catch), methodology+hyperparameters auto-reference (Step 9a-quater), lr-misprint reconcile (Check 16), auto-respawn of alive-but-stalled autonomous sessions, RunPod INSUFFICIENT_BALANCE treated as transient, fleet-burn live recompute, flash-attn in bootstrap, free-analysis-followup auto-run before parking, and more. The in-session protocol drained the workflow-fix-candidate channel almost completely.

Applied workflow improvements

Only two residual gaps survived the in-session protocol; both are additive gotchas.md bullets, lint-checked (workflow_lint.py --check-references PASS) and pushed to main. Revert either with git -C ~/explore-persona-space revert <sha>.

1. Target: .claude/rules/gotchas.md — what: verify a pod is idle via the live RunPod API + nvidia-smi GPU utilization before stopping it, never from SSH-reachability or marker-absence — commit: d0e7383e7 Why: Session 0ccdc496 [main] — Thomas: "Did you stop this: The #488 diagnostic session is still running…" → assistant: "Yes, I almost certainly killed it… a diagnostic emits no run-launched marker… SSH refused meant the cached address was stale, not that the pod was idle." A live diagnostic training run on pod-488 was killed because SSH-refused + absent run-launched marker were misread as "idle." Applied edit:

   + - **Before stopping/terminating a pod you believe is idle/done, verify with the live RunPod API + `nvidia-smi` GPU utilization — never infer "idle" from an SSH-unreachable pod or an absent `epm:run-launched` marker.** Diagnostic and manual runs emit no `run-launched` marker, and an SSH refusal usually means the cached host/port drifted (a `pod.py resume` changes them), NOT that the pod is free — both signals falsely read "nothing running." Run `pod.py config --sync` to repair any address drift, then confirm ~0% GPU utilization across ALL devices via `nvidia-smi` before any `pod.py stop` / `terminate`. (Incident #488, 2026-06-08…)
   

2. Target: .claude/rules/gotchas.md — what: document the MooseFS/FUSE D-state "zombie" failure mode + the standing off-FUSE mitigation for write-heavy runs — commit: ab22c6187 Why: Session 6ce09e09 [issue-505] — "Stop+resume pod-505 (drain the state-D zombie again)… Relaunch with EPM_OUTPUT_ROOT=/tmp/issue_505_results + WANDB_MODE=offline (route trainer writes + wandb tempfiles off FUSE)"; 88 zombie/drain mentions in one session. gotchas.md covered the EDQUOT quota but not the D-state-zombie symptom or this mitigation. Applied edit:

   + - **MooseFS/FUSE D-state "zombie" processes on write-heavy runs.** Trainer + WandB writes hammering the FUSE mount can wedge processes in uninterruptible-sleep (`D` state) that even `kill -9` cannot reap; the GPUs sit idle while the run looks alive, and the only drain is a pod `stop` + `resume`. Standing mitigation for write-heavy runs: route trainer output AND WandB tempfiles OFF the FUSE mount — `EPM_OUTPUT_ROOT=/tmp/issue_<N>_results` + `WANDB_MODE=offline` (sync WandB afterward). (Incident #505, 2026-06-08…)
   

Other problems & notes

• #506 single-GPU-stuck recovery written but never executed. The autonomous #506 session wrote a "GPU-5 stuck → drop to 7 GPUs" recovery plan, then went alive-but-stalled and never ran it; a stop+resume drifted the SSH port without re-sync and the 8×H100 idle-burned ~10.5h. The alive-but-stalled respawn itself landed today (dab26e6e1), but executing the written GPU-fallback did not. NOT auto-applied: the sweep suggested experimenter.md, but that agent is explicitly forbidden from stopping/resuming pods — the correct home is the /issue orchestrator's autonomous recovery playbook. Suggested action: add a single-rank-stuck recovery recipe to .claude/skills/issue/SKILL.md autonomous behavior (stop+resume → pod.py config --sync → relaunch with --gpu-count minus the stuck rank) — orchestrator-behavior change, route through /issue-skill review, not a blind daily auto-apply.
• #507 bootstrap re-bootstrap fails on shallow clone. bootstrap_pod.sh's existing-repo path fails with Cannot force update the current branch on a --depth=1 re-bootstrap (only the fresh-init path is hardened). Out of daily scope (shell-orchestration script, not in the allowed-target list). Suggested action: route to implementer — rewrite the existing-repo path to git fetch --depth=1 origin $BRANCH && git reset --hard FETCH_HEAD, mirroring the fresh-init strategy.
• #524 reuse-fitness check passed a statistically undersized eval pool. The reuse-fitness check approved a 50-question/cell ΔG eval pool for a correlational leakage DV; Thomas had to cancel/stop/restart #524 three times in one evening to settle the metric + pool size (50→500→100). LOW confidence on the fix, so logged not applied. Suggested action (for review): extend the planner §5 reuse-fitness check so a reused EVAL/TEST pool must also clear a statistical-power floor for the DV, not just recipe-match — flag an undersized held-out pool as a fitness failure the same way a saturated adapter is.
• Reconciler caught two real bugs the Claude reviewer PASSed — JS-reducer multi-response bias (#522) and a cross_arm_aggregator missing-flags bug (#518), both correctly flagged by the Codex twin and bound by the reconciler. The ensemble worked exactly as designed; no workflow gap. Observation only: the Claude code-reviewer twin under-covers multi-response / missing-flag cases, and the Codex ensemble is the intended net (it fired both times).
• Operational incidents handled in-session (pattern-tracking, not open fixes): #506 pod stalls from supply + INSUFFICIENT_BALANCE (~10.5h idle burn before fix); two parallel #488 sessions had to be merged; #503 burned 6 implementer rounds + 5 launch attempts on a phantom HF-reuse assumption; #489 had a 50× lr misprint (2e-6 vs 1e-4) in its Reproducibility table AND generated non-in-voice ICL examples; "zombie" approved-but-not-dispatched sessions. Each produced a committed workflow fix today.
• Data-gen-via-Claude-Code-Haiku rule (flagged as "dropped from yesterday") was deliberately not re-added. In the 3-fix greenlight list, Thomas led item 1 (the data-gen rule) with "We don't want to do this anymore" and said "Do all these" only for items 2–3, so item 1 was rejected, not missed. Resolved decision, not an open gap.

My thoughts

Highlighted results

• no results promoted today