Daily — 2026-06-25 (autonomous theory-program day;

What happened

• The autonomous overnight leakage-predictor assumption-testing program drove the whole day (sequenced /issue --auto rollout, Thomas asleep then research-Q&A by day). Four tasks ran in parallel: #664 (Phase 2 fine-tune fleet + trained store + ground-truth leakage) recovered from a pod code-sync ImportError + a build-data crash + two CUDA OOMs and is now training healthy (GPU 91%, pid 29963, ~3.5h alive); #658 (base-model foundation) — Betley judge completed 04:03Z, G1 UltraChat judge launched after download; #653 (3 behaviors × 2 contexts × 3 LoRA ranks install-validated rank ladder) actively dose-laddering on a 4×H100 pod (verified alive, 937 min CPU); #667 (leakage-predictor gate chain) reached awaiting_promotion.
• #667 clean-result parked (MODERATE): "the gate chain holds in activation space — a base-context whitened gate predicts where a fine-tune's write lands as well as the post-fine-tuning model does, but the base read-out fails and the gate only partly reaches behavioral leakage." PR #488 merged (13 issue-667 commits), methodology doc generated, follow-up #670 parked-redundant.
• Two infra tasks completed: #663 (harden Anthropic batch-judge client → ~8k/batch sharding) and #668 (shard i528_phase4_judge.py --backend batch submit).
• Workflow change shipped in-session (Thomas request): clean-result format redesigned to v4 (single document, no separate methodology section — methodology folded into the body) + low-level-data-plot rule (Lens 11: per-unit plot behind every aggregate) added to SPEC; both applied by subagents during the day. Dashboard data-viewer (Phase 2) + figure per-point sidecars also built.
• #661 halted → blocked on a deterministic GCP batch-judge poll wedge; generation is safe on HF, so judge+extraction+divergence are re-dispatchable.

Applied workflow improvements

• no workflow-fixable problems found today

Tonight's sweep found no derivable, verify-tonight workflow OR code fix to auto-apply. The day's workflow requests were already applied in-session by their own subagents/workflow-improver (the v4 clean-result redesign, the low-level-plotting Lens-11 rule, and the #653 experimenter pre-launch git-sync fix — workflow-improver af3cce3d23bce5b60, target .claude/rules/gotchas.md + experimenter agent-memory). Repo-wide workflow_lint --check-references and --check-batch-judge-client both PASS at baseline. Every remaining genuine problem is an infra/compute failure that needs pod-level diagnosis or compute spend (the judgment-call carve-out) — logged below, not auto-fixed.

Other problems & notes

• #661 deterministic batch-judge poll wedge (blocked) — HIGH. The session wedged twice polling the Anthropic Batch API for the judge-filter (2 watcher auto-respawns both re-wedged), idling a GCP A100; halted to bound burn. Repo-wide batch-judge-client lint PASSES (no hand-rolled poller), so the wedge is in the GCP-instance/networking layer, not a stray un-hardened poll. Held: judgment call (carve-out: spends money / launches compute). Suggested action: re-dispatch #661's judge+extraction+divergence on RunPod reusing the safe HF generation (issue661_rb_extraction_divergence/raw_completions) — gives an SSH-able pod to actually diagnose the wedge.
• #664 recurring CUDA OOM across phases — MEDIUM. OutOfMemoryError: Tried to allocate 39.85 GiB fired twice (18:35Z, 02:28Z) because a prior phase's vLLM engine still held 41–57 GiB when the next phase allocated. The run recovered after relaunch and is healthy now. Held: not verifiable tonight (needs the pod). Suggested action: confirm _reap_vllm_engine (now present at representation_shift.py:166) is actually called between all phases of the 664 dispatch before the next phase loads a model — the OOM signature is the known vLLM-teardown gotcha.
• #667 / #661 "hung-RUNNING VM" GCP networking wedge — MEDIUM, already tracked. Neither the sync nor async GCP→RunPod failover fires for a VM that hangs while still RUNNING (guest DHCPv4 loss freezes eps/phase at a non-terminal value), so recovery is a manual --backend runpod pivot (#667 did exactly this at 12:20Z). Per CLAUDE.md this is a pending kind: infra follow-up (poller should escalate a frozen non-terminal phase + drain-timeout to a terminal wedged state). No new action — already documented.
• #640 GCP-diagnostic wall — MEDIUM, recurring. IAP "not-authorized" blocks SSH into eps-issue-* GCE instances, so GCP crashes/wedges (the root of the #661 and #667 pain) can't be diagnosed in place; the boot disk is then DELETEd on teardown. Held: needs Thomas (account-level IAP/SSH auth on the eps-gcp project).
• #653 stale-pod-alert false positive — LOW. The watcher's >6h progress-marker-gap heuristic fired twice (14:33Z, 21:33Z) on a genuinely-busy 4×H100 select_checkpoint dose-ladder loop (verified alive: log mtime current, vLLM mid-load, 937 min CPU). Watcher correctly did NOT auto-stop — cosmetic alert noise only. Possible future fix: have long checkpoint-laddering loops post a lightweight per-step heartbeat marker so the gap heuristic doesn't false-positive. Noted; not worth a fix tonight.

My thoughts

Highlighted results

• no results promoted today (#667's MODERATE-confidence gate-chain clean-result is parked at awaiting_promotion — promote with task.py promote 667 useful|not-useful)