EPS
← All docs·Activity

Daily — 2026-06-30 (35 tasks created, ~33 merged; big autonomous workflow-fix day)

updated: 2026-07-01

What happened

  • Huge autonomous workflow-fix day. ~35 tasks created and ~33 merged to main, almost all kind: infra workflow-fixes (#741–#777) running the full planner → critic ensemble → implementer → Claude+Codex code-review → Step 10d auto-merge pipeline. Highlights that landed: #745 (HF upload acceleration — Xet HP + hf_transfer), #736 (GCP FLEX_START TimeoutExpired → still-provisioning), #741 (GCP max-run-duration 24h→7d), #743 (8× A100/H100 GCP lanes), #747 (cheap CPU-only pod lanes), #750 (_eps_phase done gated on OOM), #760 (Codex daemon reaper), #765 (the 20-guideline llm-judging.md rule), #766 (belief.py drop-to-nan), #770/#775 (RunPod no-port/CUDA-IMA wedge auto-failover), #773 (protect active-task caches), #771 (#681 data-disk bind warn).
  • Experiments in flight / landed: #658 persona-vectors-style rB round-4 merged (after 3 upload-failure retries v14–v16); #665 EM/marker Phase B+C clean-result PASS (killed+fixed a judge double-brace parse bug that cost ~27% of judge calls); #697 rescue clean-result PASS but Step-10d parked for a manual rebase (3771 commits behind); #744 clean-result merged (#568); #722 marker-predictor follow-ups (base-skill-over-mean, input-PCA-robustness) — but the task folder got corrupted (see below).
  • Rough sessions: #742 blocked after 4 consecutive production launches failed with 4 different root causes (plan-defect re-plan, ENOSPC, missing input data, SIGTERM); #734 CUDA-OOM retry chain (partial Phase-1 delivered); #763 GPU-wedged + no-compute retries; #666 three infra respawns (OOM on undersized hosts).
  • No clean-results promoted today (included_tasks empty; #744/#658/#665/#697 sit at awaiting_promotion/completed without a promoted_at on 2026-06-30).

Applied workflow improvements

  1. Target: .claude/rules/llm-judging.md — what: corrected rule-9 stale parenthetical (belief.py coerce-to-50 was FIXED in #766, now drops to math.nan) — commit: 34f8e23f38 Why: batch05 miner + verification — belief.py lines 179–181 explicitly cite "the #766 default-to-50 bug" as fixed, but llm-judging.md:87 still said "eval/belief.py currently default-coerces a malformed return to 50 — a known violation; filed as a separate code-fix." Applied edit: 
    -   the per-arm dropped count is REPORTED. (`eval/belief.py` currently
-   default-coerces a malformed return to 50 — a known violation; out of this
-   rule's workflow surface, filed as a separate code-fix.)
+   the per-arm dropped count is REPORTED. (`eval/belief.py` previously
+   default-coerced a malformed return to 50 — FIXED in #766: parse
+   failures / out-of-range / API errors now drop to `math.nan`, excluded
+   from the aggregate by the caller's `not math.isnan` filter.)
    Repo-wide lint gate (workflow_lint.py --check-references + --check-asks) PASS after the edit; pushed to main.

Filed for review (route 2 — behavior/logic changes → full /issue --auto pipeline; tagged daily-auto-filed): 2. filed for review #785: worktree-path #506 regression in Step 4a + Step 9c (--show-toplevel doubles the path; the --git-common-dir fix exists only in Step 10d) + silent test-gate cd-failure (no tests ran → exit 0). Bit 3 sessions (743/745/776); 745 silently passed the gate. 3. filed for review #786: task-workflow API atomicity — atomic status-transition git mv, typed missing-body error (not a bare FileNotFoundError), atomic post-marker events.jsonl append, view --json SIGPIPE guard. (#722 was genuinely corrupted this day.) 4. filed for review #787: Step 10d tasks/* merge-conflict default — make scratch-worktree-at-origin/main the default for far-behind branches, merge=union for events.jsonl, exclude Step-5a-synced workflow files from Guard-3. (Dominant friction, 6 batches.) 5. filed for review #788: reviewer sandbox read-only tempdir silently skips pytest (set writable TMPDIR/UV_CACHE_DIR; flag "tests NOT run" instead of PASS). Dangerous — silent test skip. 6. filed for review #789: Codex twin false-FAIL discipline — HF-DNS-in-sandbox → sandbox-unverifiable (non-blocking), and evidence-quote requirement so a hallucinated "fix applied" is self-caught before the reconciler. 7. filed for review #790: GCP fetch_results/log Permission denied (root-owned pod dirs) → sudo/chmod fetch; stop hard-FAILing confirm_artifacts on analyzer-generated figures. 8. filed for review #791: backend_poll stale-log false marker-gaps (tail newest log by mtime) + poll error-regex 0 failed false-positive. 9. filed for review #792: experiment-implementer crash-fix scope guard (fix + declare fix-engaged only; no self-relaunch, no lifecycle/failure markers). 10. filed for review #793: SLURM lane rsyncs from main, cannot honor repo_branch='issue-N' → rsync from the invoking worktree path. 11. filed for review #794: HF upload_dataset 504 verify-side false-fail (upload succeeded, tree-listing 504 → "failed silently") → tolerate/retry the verify 504. 12. filed for review #795: zombie Happy sessions on completed tasks not auto-reaped (698/705/706) → tighten the watcher reconcile predicate. 13. filed for review #796: guard detached repo-root HEAD (crashes the task_workflow resolver; the guard hook covers branch-switch but not detach). 14. filed for review #797: routing carve-out — interactive 0-GPU inline analysis Thomas asks to run NOW should run inline with a subagent, not be pushed through the follow-up loop (Thomas correction). 15. filed for review #798: artifact-reuse (h) check must assert reused files resolve at the EXACT dispatcher-asserted paths, not just repo/dir existence (#474 vs #664 naming mismatch crashed Phase 2). 16. filed for review #799: agent output-format — planner must emit markdown plans (not .html); paper-plots must carry NO interpretive text in figures (Thomas corrected #722/#771 plots). 17. filed for review #800: document the WANDB_LOG_MODEL HF-Trainer checkpoint→WandB upload trap in upload-policy.md (guard is EPM_UPLOAD_MODEL_WANDB=1, default off).

Other problems & notes

  • Route 3 (needs-human) — filed #801: the 5 registry/filesystem inconsistencies (living_docs check: #696/#722/#750/#764/#766) need a human-audited repair. Held under the judgment-call carve-out (destructive/state mutation + genuinely ambiguous): #722 is a KNOWN real corruption (a non-atomic git mv split it across two folders with no REGISTRY entry), so blindly running task.py audit could repoint the registry at the stub and lose the real 27KB body + plans v1–v7. Confirm which folder holds the real data first. Structural cause filed as #786.
  • Repo hygiene (applied): removed a stray 0-byte junk file **Figure.** (a botched-redirect artifact, mtime 07:39) that polluted every session's git status. Not a workflow-file edit, so recorded here, not in Applied improvements.
  • Recurring agent-discipline slips (rules exist — not filed): bare python in a pipe (blocked by the PreToolUse hook, self-recovered); cd "$(task.py find N)" persisting into the next Bash call and breaking the marker post; $REPO_ROOT used in a fresh Bash call where it isn't exported (expands empty → /scripts/task.py); git rev-parse --show-toplevel returning the worktree root inside a worktree (doubled path). All are documented; the #506 case with an INCOMPLETE fix is filed as #785, the rest are discipline slips already covered.
  • Already covered by tasks filed/merged today (not re-filed): hf_transfer (#745), _eps_phase done OOM mask (#750), GCP FLEX_START timeout (#736), Codex daemon leak (#760), belief.py coerce (#766), duplicate per-issue session respawn (#759), failover pod sync (#751), watcher no-port wedge (#770/#775). The watcher proposed-infra-sweep firing multiple times (batch08) overlaps #759.
  • Cosmetic: every EPS uv run prints VIRTUAL_ENV=/home/.../introsp/.venv does not match — a stale env var leaking from the sibling introsp project (uv ignores it; noise only). Consider unset VIRTUAL_ENV in the EPS profile.
  • Ops (tracked, not new): / hit 99% and /mnt/eps-data is still absent — the deferred #681 data-disk bind-migration; disk guard fired as designed. Tracked in memory project_vm_data_disk_not_attached.md.
  • Working-as-intended (positive): the review ensembles caught real bugs (implementer-brief-vs-plan contradictions, hallucinated fixes) before merge; the #722/#665 sessions correctly declined to mutate corrupt state / killed-and-fixed rather than finishing on bad data; the single-orchestrator guard yielded on cross-dispatched collisions.

Living-docs drift

  • living_docs.py check FAIL — 202 findings (drift hash 0c54a0b92aed): 5 registry/filesystem inconsistencies (#696/#722/#750/#764/#766, surfaced as #801 above); ~79 relates_to '<anchor>' but no question with that anchor exists; ~118 completed clean-results appearing in no question's evidence.
  • Chronic re-synthesis proposal already OPEN (2026-06-29, hash 1736fca0804f, unreconciled — open_questions.md changelog block is empty). Today's hash differs only because the orphan set grew + the 5 new registry/fs findings; the re-synthesis it asks for would reconcile today's superset too. Not re-proposing (the open proposal covers it); today's superset recorded to the dedup event stream.
  • Parked-proposal re-consideration (Step B): no newly-actionable revivals — the on_hold set is the legacy idea backlog (untagged), and no still-open parked-redundant follow-up or declined living-docs rejection had a context-change trigger this run.

My thoughts

Highlighted results

  • no results promoted today