EPS
← All docs·Activity

Daily — 2026-06-28 (workflow-automation push lands;

updated: 2026-06-29

What happened

  • Workflow-automation push landed answering today's repeated "I barely run /daily, /weekly — automate this": #711 (deterministic consolidate_lessons.py cron + daily healthcheck), #713 (fold /weekly's load-bearing checks into nightly /daily, retire the weekly cadence), and the #706 three-route /daily classifier all completed. /daily no longer "grades its own homework" — behavior changes now route to independent review.
  • Six workflow-fix infra tasks completed: #716 (task.py set-status git-mv on destination collision), #718 (crash-fix circuit-breaker → force re-plan once a fallback ladder is spent — the #664 lesson), #719 (per-cell vLLM rendezvous-port EADDRINUSE gotcha), #720 (reap completed autonomous /issue sessions in ~1h, not 12h — the ghost-session fix), #721 (/daily push stub immediately so it isn't orphaned off main, #711 class), #723 (rsLoRA parity probe needs A100 intent).
  • #664 (leakage Phase 2 fine-tune fleet) fought a long saga — reproducible vLLM generate() hang + zombie-GPU allocation, CUDA OOM on p1, and a marker read-gauge HALT — across multiple recovery rounds; still running. The save-and-resume worked; the time burned was a real vLLM bug inside the not-yet-checkpointable p0 step.
  • New theory-driven experiments launched/discussed: #715 (does DFT attenuate EM vs standard SFT — tried GCP twice, both crashed, correctly failed over to RunPod), #722 (fit M0 vs M⁺ context→answer mapping on #537 adapters, ridge + MLP), #697 (causal context-vector patch on #537 — does finetuning shift the context vector or the mapping?). #667 leakage-predictor gate-chain clean-result sits at MODERATE confidence (followups_running).
  • No clean-results promoted today.

Applied workflow improvements

No route-1 (trivial mechanical, no-behavior-change) fixes tonight — the day's workflow-fixable problems were already caught and fixed in-flight via the proper /issue pipeline (#716/#718/#719/#720/#721/#723). The three problems found by tonight's sweep are all behavior changes → filed for independent review (route 2):

  1. filed for review #724 (route 2): task.py audit is report-only and found 13 registry/filesystem inconsistencies it can't reconcile — #696 registry path says approved/696 but the task is at completed/696, and #698–709 are on disk but absent from REGISTRY.json (so invisible to the dashboard list + every registry-reader; terminal tasks never self-heal). Filed to add a task.py audit --repair / registry-rebuild and reconcile the current drift. Target: scripts/task.py, src/explore_persona_space/task_workflow.py. Likely a concurrent-rebase-flattened registry update (#711 class).
  2. filed for review #725 (route 2): living_docs.py check — now a load-bearing nightly /daily step (#713) — hard-crashes with FileNotFoundError on the single drifted task #696, silently killing the entire nightly drift pass (it produced no drift report tonight). Filed to make it skip+warn the bad task and surface it as a drift finding instead of raising. Target: scripts/living_docs.py.
  3. filed for review #726 (route 2): the Happy initial-prompt injection is a sudo monkey-patch of a vendored Happy file (patch_happy_daemon.py, sentinel v4) that silently reverts on npm update happy; after a revert, spawn_session.py injection no-ops and spawned sessions sit idle without firing /issue N (the latent residual behind today's idle-session pile; #720 fixed the reaping, not this). Dormant today but guaranteed to bite on the next Happy upgrade. Filed to have the spawn path run the patch's --check (re-apply / fail-loud) before relying on injection. Target: scripts/spawn_session.py, scripts/patch_happy_daemon.py. Surfaced in-session during the #720 deep-check (session 3cd355aa).

Other problems & notes

  • #715 "why didn't it use GCP, isn't GCP first resort?" — answered, working as designed: the router tried GCP FLEX_START A100-80 twice (attempts #1 and #2), both crashed as workload failures, and it correctly failed over to RunPod per the #658 GCP→RunPod workload-failover policy. Not a bug.
  • #664 vLLM hang/zombie-GPU/OOM saga — handled in-flight (#718 circuit-breaker + #719 rendezvous-port gotcha landed; failure-lessons posted). The recovery loop's lack of a forced re-plan once the fallback ladder was spent was the workflow gap — now closed by #718. #664 still running; no further /daily action.
  • Codex probe "No job found for task-…" ×28 (notably #697, #714) — transient Codex job-registry lag; each "exhausted 1 transient re-dispatch" so it's bounded, not a stuck loop. Watch; if the volume keeps climbing, the codex_task.py probe re-dispatch budget may be worth raising (not filed — speculative tuning, not a clear bug).
  • Bare python slips (~6 real command not found, exit 127) — the documented gotcha (VM has no python symlink; use uv run python). Self-correcting; already in CLAUDE.md + memory. No action.
  • ModuleNotFoundError: runpod_api ×45 / …experiments.issue503 ×26 — the former is in-flight tests/test_issue_dispatch.py / test_backend_poll_script test-isolation work (an agent diagnosed it as a missing self-sufficient sys.path import, being fixed within its task); the latter's module exists on disk (per-run path/editable-install issue during a run). Both handled within their tasks; not workflow-surface.
  • Registry drift data repair (#696 stale + #698–709 unregistered) — the repair itself is task-state and is routed through the reviewed pipeline via #724 rather than a /daily hand-edit of REGISTRY.json.
  • Two month-old zombie Happy wrappers in ~/my-goat (pids 1826152 / 1826434, alive ~30 days, no inner Claude) — OUT of EPS scope (the EPS reaper correctly skips non-EPS cwd); a my-goat hygiene leak. Safe to kill, Thomas's call.
  • 429 rate-limit hits ×6 — normal token pacing, climbs at the minute boundary. No action.

Living-docs drift

  • Drift could not be assessed tonight: living_docs.py check crashed on the stale #696 registry path (FileNotFoundError) before producing any drift report. The crash itself is filed for review as #725 (graceful-degrade) and the underlying registry drift as #724 (repair). The parked-proposal re-consideration pass (Step B) is deferred this run — it depends on the same registry resolution path that crashed. Re-run after #724/#725 land to get a clean drift report.

My thoughts

Highlighted results

  • no results promoted today