Daily — 2026-06-28 (workflow-automation push lands;
updated: 2026-06-29
What happened
- Workflow-automation push landed answering today's repeated "I barely run /daily, /weekly — automate this": #711 (deterministic
consolidate_lessons.pycron + daily healthcheck), #713 (fold /weekly's load-bearing checks into nightly /daily, retire the weekly cadence), and the #706 three-route /daily classifier all completed. /daily no longer "grades its own homework" — behavior changes now route to independent review. - Six workflow-fix infra tasks completed: #716 (
task.py set-statusgit-mv on destination collision), #718 (crash-fix circuit-breaker → force re-plan once a fallback ladder is spent — the #664 lesson), #719 (per-cell vLLM rendezvous-portEADDRINUSEgotcha), #720 (reap completed autonomous /issue sessions in ~1h, not 12h — the ghost-session fix), #721 (/daily push stub immediately so it isn't orphaned off main, #711 class), #723 (rsLoRA parity probe needs A100 intent). - #664 (leakage Phase 2 fine-tune fleet) fought a long saga — reproducible vLLM
generate()hang + zombie-GPU allocation, CUDA OOM on p1, and a marker read-gauge HALT — across multiple recovery rounds; stillrunning. The save-and-resume worked; the time burned was a real vLLM bug inside the not-yet-checkpointable p0 step. - New theory-driven experiments launched/discussed: #715 (does DFT attenuate EM vs standard SFT — tried GCP twice, both crashed, correctly failed over to RunPod), #722 (fit M0 vs M⁺ context→answer mapping on #537 adapters, ridge + MLP), #697 (causal context-vector patch on #537 — does finetuning shift the context vector or the mapping?). #667 leakage-predictor gate-chain clean-result sits at MODERATE confidence (followups_running).
- No clean-results promoted today.
Applied workflow improvements
No route-1 (trivial mechanical, no-behavior-change) fixes tonight — the day's workflow-fixable problems were already caught and fixed in-flight via the proper /issue pipeline (#716/#718/#719/#720/#721/#723). The three problems found by tonight's sweep are all behavior changes → filed for independent review (route 2):
- filed for review #724 (route 2):
task.py auditis report-only and found 13 registry/filesystem inconsistencies it can't reconcile —#696registry path saysapproved/696but the task is atcompleted/696, and#698–709are on disk but absent fromREGISTRY.json(so invisible to the dashboard list + every registry-reader; terminal tasks never self-heal). Filed to add atask.py audit --repair/ registry-rebuild and reconcile the current drift. Target:scripts/task.py,src/explore_persona_space/task_workflow.py. Likely a concurrent-rebase-flattened registry update (#711 class). - filed for review #725 (route 2):
living_docs.py check— now a load-bearing nightly /daily step (#713) — hard-crashes withFileNotFoundErroron the single drifted task#696, silently killing the entire nightly drift pass (it produced no drift report tonight). Filed to make it skip+warn the bad task and surface it as a drift finding instead of raising. Target:scripts/living_docs.py. - filed for review #726 (route 2): the Happy initial-prompt injection is a
sudomonkey-patch of a vendored Happy file (patch_happy_daemon.py, sentinel v4) that silently reverts onnpm update happy; after a revert,spawn_session.pyinjection no-ops and spawned sessions sit idle without firing/issue N(the latent residual behind today's idle-session pile;#720fixed the reaping, not this). Dormant today but guaranteed to bite on the next Happy upgrade. Filed to have the spawn path run the patch's--check(re-apply / fail-loud) before relying on injection. Target:scripts/spawn_session.py,scripts/patch_happy_daemon.py. Surfaced in-session during the #720 deep-check (session3cd355aa).
Other problems & notes
- #715 "why didn't it use GCP, isn't GCP first resort?" — answered, working as designed: the router tried GCP FLEX_START A100-80 twice (attempts #1 and #2), both crashed as workload failures, and it correctly failed over to RunPod per the #658 GCP→RunPod workload-failover policy. Not a bug.
- #664 vLLM hang/zombie-GPU/OOM saga — handled in-flight (#718 circuit-breaker + #719 rendezvous-port gotcha landed; failure-lessons posted). The recovery loop's lack of a forced re-plan once the fallback ladder was spent was the workflow gap — now closed by #718. #664 still running; no further /daily action.
- Codex probe "No job found for task-…" ×28 (notably #697, #714) — transient Codex job-registry lag; each "exhausted 1 transient re-dispatch" so it's bounded, not a stuck loop. Watch; if the volume keeps climbing, the codex_task.py probe re-dispatch budget may be worth raising (not filed — speculative tuning, not a clear bug).
- Bare
pythonslips (~6 realcommand not found, exit 127) — the documented gotcha (VM has nopythonsymlink; useuv run python). Self-correcting; already in CLAUDE.md + memory. No action. ModuleNotFoundError: runpod_api×45 /…experiments.issue503×26 — the former is in-flighttests/test_issue_dispatch.py/test_backend_poll_scripttest-isolation work (an agent diagnosed it as a missing self-sufficientsys.pathimport, being fixed within its task); the latter's module exists on disk (per-run path/editable-install issue during a run). Both handled within their tasks; not workflow-surface.- Registry drift data repair (#696 stale + #698–709 unregistered) — the repair itself is task-state and is routed through the reviewed pipeline via #724 rather than a /daily hand-edit of
REGISTRY.json. - Two month-old zombie Happy wrappers in
~/my-goat(pids 1826152 / 1826434, alive ~30 days, no inner Claude) — OUT of EPS scope (the EPS reaper correctly skips non-EPS cwd); a my-goat hygiene leak. Safe to kill, Thomas's call. - 429 rate-limit hits ×6 — normal token pacing, climbs at the minute boundary. No action.
Living-docs drift
- Drift could not be assessed tonight:
living_docs.py checkcrashed on the stale#696registry path (FileNotFoundError) before producing any drift report. The crash itself is filed for review as #725 (graceful-degrade) and the underlying registry drift as #724 (repair). The parked-proposal re-consideration pass (Step B) is deferred this run — it depends on the same registry resolution path that crashed. Re-run after #724/#725 land to get a clean drift report.
My thoughts
Highlighted results
- no results promoted today