Daily — 2026-06-26 (clean-result paper-mode redesign; GCP capacity + VM-disk firefighting; 3 fixes auto-applied)

What happened

• Clean-result format → LaTeX paper-mode (#657): Thomas drove a multi-step redesign — drop the standalone methodology doc, fold it into Goal/Methodology/Results, require low-level labeled-scatter data plots, demand self-contained methodology (reused artifacts get their recipe rewritten inline, not "reused from #N"), and land an opt-in full LaTeX paper format with verbatim examples + judge prompts. Shipped as verify_paper.py checks 7-8 + the v4 spec.
• #653 (install-validated re-ladder, round 6): relaunch crashed _ablation_gpu_read with FileNotFoundError — phase_ablation iterated all behaviors with no drop-gate while phase_dx legitimately skipped below-floor em cells. Fixed in-session (drop-gate + resume-skip), relaunched, parked at awaiting_promotion. Also hit the pod-sync silent-.git/index.lock HEAD-not-advancing trap (caught at the HEAD-verify step; failure-lesson + gotchas entry persisted in-session).
• VM root disk to 100% again: #658's Phase-1 analysis materialized a ~140 GB activation store on the shared CPU-only VM worktree. Thomas: "why is 658 writing to our CPU only VM?" → drove the cpu-bigmem GCP lane (#677) + disk-guard work (#679, lead structural follow-up #681). The disk-guard's only HF-cache reclaim no-op'd at CRITICAL (scanned the wrong cache dir) — fixed tonight (below).
• GCP keeps failing (networking wedge, ZONE_RESOURCE_POOL_EXHAUSTED, bootstrap hang): #680 (spot-first / flex-start length-aware router ladder) was planned, live-probed, and merged to improve capacity. The hung-but-RUNNING wedge (#667) stays a manual RunPod pivot.
• #664 dispatcher crashed 3× on HF Hub 429 — per-probe AutoTokenizer.from_pretrained triggered a model_info() HTTP call each load. Fixed in-session (module-level tokenizer cache); generalized to a gotchas entry tonight.
• #678 (retire the workflow-improver auto-spawn → file kind:infra + /issue --auto) merged; #683 / #685 (behavior-implant + context experiments) iterated through the review ensemble.

Applied workflow improvements

1. Target: scripts/autonomous_session_watch.py — what: point the disk-guard HF-hub-TTL eviction at the real cache dir (HF_HUB_CACHE / HF_HOME/hub) instead of the unset default — commit: 8ef21b9f76 Why: at VM-disk CRITICAL the only HF-cache reclaim no-op'd: hf-hub-ttl skipped (scan failed: Cache directory not found: /home/thomasjiralerspong/.cache/huggingface/hub. Please use cache_dir argument or set HF_HUB_CACHE) (#658 watcher vm-disk pass) — the VM redirects the HF cache, so a bare scan_cache_dir() looked at the wrong path and reclaimed 0.0 GiB at the worst moment. Verified: syntax OK + scan_cache_dir accepts cache_dir. Applied edit:

   +            _hub_cache = os.environ.get("HF_HUB_CACHE") or (
   +                os.path.join(os.environ["HF_HOME"], "hub")
   +                if os.environ.get("HF_HOME") else None)
   -                cache_info = scan_cache_dir()
   +                cache_info = (
   +                    scan_cache_dir(cache_dir=_hub_cache) if _hub_cache else scan_cache_dir())
   

2. Target: CLAUDE.md (§ Task Workflow API) — what: two recurring orchestrator one-liner traps documented — commit: da5ceb19ab Why: (a) python: command not found exit 127 from uv run python scripts/task.py view 658 --json | python -c "…" — the consumer side of a pipe used bare python (#658); the "use uv run python" rule lived only in auto-memory, not project CLAUDE.md. (b) task.py post-marker: error: argument --note: not allowed with argument --file exit 2 — the alternation wasn't called out in prose (#653). Applied edit:

   - post-marker <N> epm:foo --note '...'   # long body: --file <path.md> (the flag is --file, NOT --note-file)
   + post-marker <N> epm:foo --note '...'   # ... --note and --file are MUTUALLY EXCLUSIVE — pick one (passing both raises argparse error exit 2).
   + **Bare `python` is unavailable on this VM** — prefix EVERY python invocation with `uv run python`, INCLUDING the consumer side of a pipe (`... | uv run python -c "..."`, NEVER `... | python -c "..."`).
   

3. Target: .claude/rules/gotchas.md — what: new gotcha: per-probe AutoTokenizer.from_pretrained → silent per-load model_info() HTTP → HF Hub 429; cache the tokenizer at module scope — commit: 99b231580e Why: #664 dispatcher crashed 3× — _prompt_text_for (eval line 385) called AutoTokenizer.from_pretrained per probe; newer transformers runs a model_info() Hub call inside _patch_mistral_regex every load → 429 after ~16 cells. The code was fixed in-session (issue664_eval.py module-level cache); the per-call-from_pretrained → silent-HTTP → 429 pattern recurs across eval rigs, so it's generalized here.

Lint gate after all workflow-file edits: workflow_lint.py --check-references PASS, --check-asks PASS. Revert any with git -C ~/explore-persona-space revert <sha>.

Other problems & notes

• HELD — planner.md CPU/GPU-overlap rule. Thomas at 01:22: "CPU only work (e.g. judging) gets done in PARALLEL to GPU work as much as possible … we run on GPUs in parallel AS MUCH AS POSSIBLE" and "why did we run i on 1 H100 instead of in parallel on 8 H100s?" He DISPATCHED this as a background /issue session, so planner.md §9 is the live topic of an in-flight workflow edit — editing it tonight would collide with that session's rewrite (the workflow-fix "live-topic-of-an-in-flight-change" deferral). Suggested action: confirm the spawned session landed the CPU-judge-overlaps-GPU + parallelize-independent-GPU-work rule in planner.md §9; file it if it didn't.
• HELD (B — launches compute / scientific run placement). #658 placed a ~140 GB activation store on the shared CPU-only VM worktree, filling / and stalling the fleet. The durable fix (the cpu-bigmem GCP lane #677, disk-guard #679, structural quota fix #681) is already in flight; the live #658 run predated it. No auto-edit.
• HELD (B — failover semantics / spends compute, #667). A hung-but-RUNNING GCP VM (guest networking died, eps/phase frozen at non-terminal) matches neither the sync nor async failover predicate → manual --backend runpod pivot. The fix (escalate a frozen non-terminal phase + drain-timeout to a terminal wedged state, fail over to RunPod) is a real but unlanded infra decision the #672 session deliberately deferred. Suggested action: file kind:infra for the #667 escalation predicate.
• HELD (B — already corrected live). #672 was mis-filed as kind: experiment (a promotable clean-result) when it was a fix-VALIDATION → Thomas corrected it live via task.py set-kind <N> infra. The "fix-validation → kind:infra" routing rule is already in CLAUDE.md (added after this exact incident); gap closed.
• HELD (B — already fixed live). A 90k-request judge batch went out un-chunked because issue658_judge_e0_batch.py carries its own inline messages.batches.create (CHUNK=90000) bypassing the #663-hardened eval/batch_judge.py, and an autonomous session parked to "await your call" instead of taking the free harvest path. Both got live fixes (workflow_lint --check-batch-judge-client blocks NEW offenders + the autonomous-no-park-when-free-path-exists rule). Note: the existing issue658_judge_e0_batch.py + deferred i528_phase4_judge.py still carry the inline path; NOT retrofitted tonight (the run is complete and migrating completed-run judge code mid-flight is risky / scientific-meaning-adjacent).
• SKIPPED (self-correcting harness behavior, no fix warranted). Recurring File has not been read yet, SendMessage InputValidationError (dead agent id), sleep N && tail harness-block, guard_log_dump blocks on cat of large logs, and Read exceeds maximum tokens — all self-corrected in-session and the governing rules already live in CLAUDE.md. Adding reminder lines to agent specs is low-value bloat; skipped to keep the specs lean.
• NOTE (one-off, low recurrence). #683 round-3 orchestrator grepped a Codex output file before it existed (No such file or directory). Suggested action if it recurs: add an explicit "confirm epm:codex-task-completed (or file presence) before reading the Codex output file" guard to issue/SKILL.md Codex-ensemble dispatch. Single occurrence tonight — not auto-applied.

My thoughts

Highlighted results

• no results promoted today