EPS
← All docs·Activity

Daily — 2026-06-10 (40 tasks filed, ~20 parked for promotion; 16 workflow fixes auto-applied; 3 promoted)

updated: 2026-06-11

What happened

  • Highest-throughput day yet: 40 new tasks created (#551–#590), 4,850 commits on main. ~20 tasks reached awaiting_promotion during the day (the column now holds 25); 10 are running, 2 interpreting, 2 in same-issue follow-up loops (#472, #552). 3 promotions (#501, #509, #549 — all useful) and 4 user-approved retroactive same-question merges (#547→#533, #548→#540, #550→#538, #551→#521).
  • Infra landings: #535 multi-backend compute router matured (Nibi 4/4 + Mila 4/4 live-acceptance PASS, GCP lane debugged through fail-loud secrets + VM self-poweroff + branch threading, GCP-first auto lane order); #586 campaign runner (/campaign + /campaign-tick) merged; #584 completed; #587 (per-issue progress bar/ETA) dispatched; dashboard got public read-only /tasks + 60s auto-refresh after Thomas hit stale kanban columns.
  • The day's physical constraint was disk + fleet caps: VM root hit 99-100% full at least 4 separate times (276GB of worktrees, 15-16G HF caches rooted inside worktrees, a 43G uv cache — pruned twice); the worktree-audit reaper was being defeated both by zombie finished-issue Happy sessions and by a stale du scan that argv-pinned every worktree. Fixes landed same-day: zombie-wrapper reaper pass, session-reconcile auto-stop (Thomas's call, superseding his same-morning "alert-only" decision: "stop the happy sessions once they reach awaiting promotion"), watcher disk-headroom guard. Fleet $/hr cap queued 4-5 provisions at once (#537 waited ~4h).
  • Spurious Anthropic usage-policy refusals were the worst single failure class: ~25 agent kills across ≥10 sessions, including 3 orchestrator-session kills (#543 bricked ~75 min by re-refused tick re-drives; #472 and #563 sessions died outright). Analyzers ingesting raw completion text were the most common victim; every task recovered via retries/respawns, but the cost was real (one analyzer died 82 tool calls in with zero durable writes).
  • Two expensive near-misses caught in-session: #537's marker trainer silently used the deprecated [ZLT] token (all 16 adapters were no-op implants; caught before the full P1 spend, 16 cells retrained), and the HF storage-quota saga closed (4.98TB purged; #545/#546/#543 unblocked). ~50 in-session workflow-fix commits landed during the day, on top of the 16 from this sweep below.

Applied workflow improvements

All 16 fixes lint-clean (workflow_lint.py --check-references + --check-asks PASS; 154 workflow-invariant tests pass). Revert any one with git -C ~/explore-persona-space revert <sha>. A 17th commit (ada141cd8) is the auto-table regen the linter required after fix 1.

  1. Target: .claude/workflow.yaml, .claude/agents/upload-verifier.md, .claude/skills/issue/SKILL.md, .claude/agents/research-pm.md — what: retire the phantom uploading status from every workflow surface — commit: c5b32ddca (+ table regen ada141cd8) Why: at least 7 sessions across 6 sweep groups ran task.py set-status <N> uploading and crashed on the enum (invalid status 'uploading') — workflow.yaml modeled uploading as a real status and upload-verifier.md/SKILL.md prescribed setting it, while task.py's enum has only verifying. research-pm.md's "canonical" status list also carried queued/implementing/code_reviewing/testing phantoms (this fed the PM's wrong status-ownership framing Thomas corrected). Applied edit:

    - - name: uploading            (workflow.yaml § statuses block, + canonical_lifecycle + 3 entry lists)
- status:uploading does not advance to interpretation.
+ status:verifying does not advance to interpretation.
- advancement from status:uploading to status:interpreting   (upload-verifier.md)
+ advancement from status:verifying to status:interpreting
- exit loop; transition to status:uploading; go to Step 7.   (SKILL.md poll loop)
+ exit loop; transition to status:verifying; go to Step 7.
+ research-pm.md status list = the real 13-value task.py enum

  2. Target: .claude/skills/issue-tick/SKILL.md — what: refusal-thinned re-drive after a usage-policy-killed orchestrator turn — commit: ca8f8952c Why: the #543 session was bricked ~75 min on 2026-06-10 (08:54–10:08Z): the orchestrator turn died on a spurious refusal and every /issue-tick 543 re-drive replayed the same trigger-dense context and was re-refused 4×. #472's and #563's sessions died the same way. Applied edit:

    + **Refusal-thinned re-drive (applies to every re-drive in 3b AND 3c).**
+ On a refusal-killed predecessor turn, re-drive THINNED: resume from
+ `task.py latest-marker <N>` + status only — do NOT page the clean-result
+ body / interpretation / raw-completion text back in; if the thinned
+ re-drive is refused too, defer to the watcher respawn.

  3. Target: .claude/agents/analyzer.md — what: content firewall defaults ON for all safety-vocab corpora + periodic fact-sheet checkpointing — commit: efc9b9d99 Why: the #521 firewall was scoped to "harmful-content tasks"; on 06-10 analyzers on #543, #558, #562, #563, #464 were refusal-killed over corpora that didn't look harmful (key-string-prefixed military Q&A, trigger-keyed-rule framings). One #557 analyzer died 82 tool calls in with zero durable writes. Applied edit:

    - **Content firewall for harmful-content tasks (EM evals, jailbreak data, misaligned completions): never page raw-completion files into your context.**
+ **Content firewall — DEFAULT ON for every task in this project's safety-research vocabulary class (… AND marker / trigger / implant / backdoor corpora) …** 'this corpus is benign' is NOT a reason to skip the firewall. When in doubt, firewall.
+ Additionally, checkpoint your fact-sheet to `.claude/cache/` every ~15-20 tool calls …

  4. Target: .claude/workflow.yaml (§ concerns_protocol), .claude/skills/issue/SKILL.md — what: document the 200-char raise-concern --summary hard cap — commit: f339761d9 Why: five reviewer flows crashed on ValueError: summary too long in five different sessions (238/253/268/298/410-char summaries); only code-reviewer.md stated the cap, so every fresh context re-discovered it by crashing. Applied edit:

    + summary_cap: |
+   `--summary` is HARD-CAPPED at 200 chars — task.py raises ValueError
+   above it … move all detail to `--evidence`. Applies to address-concern too.

  5. Target: .claude/rules/gotchas.md — what: vLLM prompt_logprobs logits-OOM rule + name import peft in the CVD-freeze bullet — commit: 2a83d9785 Why: #501 hit two CUDA OOMs inside vLLM's log_softmax (float32 logits for a whole scheduler step at max_model_len=32768 ≈ 19 GiB; fixed by max_num_batched_tokens=8192 + gpu_mem ≤0.75, commit cb4e4119f); #545 round-10 root-caused import peft as the cuInit source that froze the device list (all +gpu_id pins silently inert → co-located OOM). The dotenv-heredoc and SSH-MCP-30s gotchas the sweep also flagged were ALREADY in gotchas.md from yesterday. Applied edit:

    + - **vLLM `prompt_logprobs` + large `max_model_len` OOMs on float32 logits materialization.** … RULE: max_num_batched_tokens≈8192 AND gpu_memory_utilization<=0.75 …
- if any import in the dispatcher chain initializes CUDA before
+ if any import in the dispatcher chain initializes CUDA (`import peft` is a known offender — incident #545) before

  6. Target: .claude/rules/research-project-structure.md — what: replace the stale setup_env() reference with the real loader + canonical inline env recipe — commit: fec699b95 Why: two sessions followed the documented setup_env() import and crashed (ImportError — no such symbol in utils.py); the real loader is load_dotenv() in orchestrate/env.py. 7+ sessions independently re-derived the set -a && source .env inline recipe after HF_TOKEN missing / find_dotenv() stack-walk failures. Applied edit:

    - Every entrypoint calls `setup_env()` from `src/explore_persona_space/utils.py`:
+ Every entrypoint calls `load_dotenv()` from `src/explore_persona_space/orchestrate/env.py` (there is NO `setup_env()` …)
+ For ad-hoc inline one-liners: `set -a && source .env && set +a && uv run python - <<'PY' … PY`

  7. Target: .claude/agents/research-pm.md — what: immediate stop+respawn on approving a stalled session; filtered session-existence checks — commit: 6cdf1eee5 Why: two user-corrected PM gaps: after approving #545 with its session known-stalled, the PM armed a 25-min background check instead of respawning (Thomas: "can't you just start it now" ×2); and the PM asserted #524 had no session from a 40-line tail of a 56-session list (it did — Thomas: "are you sure 524 doesn't have a session?"). Applied edit:

    + **Approval of a task whose owning session is stalled/dead → stop + respawn IMMEDIATELY.** … Background checks are for HEALTHY sessions only.
+ **Session-existence claims require a filtered FULL listing.** `spawn_session.py list | grep -w <N>` + watcher registry; never an eyeballed tail.

  8. Target: .claude/rules/upload-policy.md — what: resume-critical pipeline INPUTS upload before any deliberate stop-for-resume — commit: 58403d3a0 Why: #488's host-pinned resume hit SUPPLY_CONSTRAINT ~18× while the regenerated training rows + Phase 0/1 outputs + diagnostic adapters existed ONLY on the stopped pod's volume; the pod-side smoke shipped "INFRA BLOCKED, local evidence only". Extends the #521 analysis-tensors rule upstream. Applied edit:

    + **Resume-critical pipeline INPUTS must upload before any deliberate `pod.py stop` that expects a later resume.** RunPod resume is HOST-PINNED … push R_train caches / phase outputs / diagnostic adapters to the HF data repo BEFORE stopping.

  9. Target: .claude/rules/workflow-fix-on-bug.md, .claude/skills/issue/SKILL.md, CLAUDE.md — what: git pull --rebase --autostash is the documented repo-root recovery form — commit: 677187b10 Why: #555, #558, #518 all hit cannot pull with rebase: You have unstaged changes following the documented plain git pull --rebase — the shared root always carries dirty agent-memory/figure noise from concurrent sessions; each recovered ad hoc with --autostash. Applied edit:

    - git -C "$REPO_ROOT" pull --rebase && git -C "$REPO_ROOT" merge --no-ff …
+ git -C "$REPO_ROOT" pull --rebase --autostash && git -C "$REPO_ROOT" merge --no-ff …

  10. Target: .claude/skills/issue/SKILL.md — what: ban no-op Bash yield calls while bg work is in flight — commit: f060af92b Why: the #547 session issued 33× sleep 1 "Yield turn" calls and the #556 session 49× true no-ops to "stay waiting" on bg work; the harness re-invokes on bg-Bash exit regardless, so each call burned a tool call + context for nothing. Applied edit:

    + # To WAIT on bg work, simply END THE TURN with a one-sentence status — NEVER
+ # emit no-op Bash calls to idle (`sleep 1` "yield turn", `true` no-ops) …

  11. Target: .claude/rules/marker-leakage-measurement.md — what: the marker token-id assert must be wired INTO the training entrypoint (in-process, fail at startup) — commit: d07da68de Why: #537's marker trainer silently used the deprecated [ZLT] token — all 16 adapters were no-op implants, caught only after the GPU spend (16 cells retrained). The pre-spawn shell-assert convention demonstrably isn't enough; absence of the in-process assert is now an implementer-review blocker. Applied edit:

    - Launchers must assert `tokenizer.encode(…) == [83399]` before any subprocess spawns.
+ The assert must be WIRED INTO the training entrypoint / dispatcher itself … a pre-spawn shell check or convention is NOT sufficient. (Incident #537 …)

  12. Target: .claude/skills/issue/SKILL.md (Step 0) — what: worktree sessions run the spec-freshness sync BEFORE cron arming + resolve workflow scripts from the main checkout — commit: d70eba704 Why: #501's worktree pinned a pre-issue-tick-split skill copy: the session armed the heavyweight /issue 501 cron at */10 instead of /issue-tick at */20 — 362 full ~44K-token skill reloads over 2.5 days (~16M redundant input tokens). #496's worktree carried a pre-W22 verify_task_body.py that false-FAILed a conformant body. Applied edit:

    + **Worktree spec-freshness BEFORE arming.** … run the Step 5a spec-freshness sync FIRST, and resolve workflow-helper scripts from the MAIN checkout ("$REPO_ROOT"/scripts/…), never the worktree copy.

  13. Target: .claude/agents/analyzer.md, .claude/skills/issue/SKILL.md — what: verify_task_body.py snippets pin to "$REPO_ROOT"/scripts/ — commit: 14ca1af0e Why: companion to fix 12 at the copy-paste level: agents copy the verifier invocation verbatim, and the relative scripts/... form resolves to a possibly spec-stale worktree copy (incident #496). Applied edit:

    - uv run python scripts/verify_task_body.py --issue <N>
+ uv run python "$REPO_ROOT"/scripts/verify_task_body.py --issue <N>  # main-checkout copy, never the worktree's

  14. Target: .claude/rules/code-style.md — what: 529 Overloaded is a default transient in judge/API retry wrappers; per-row judge_failed over whole-launcher crashes — commit: a332ed83d Why: #556's 4,400-judgment phase crashed on ~97 untrapped 529s, and the first hot-patch guessed a nonexistent anthropic.OverloadedError symbol; #528's judge crashed three different ways on per-row anomalies (empty-response SystemExit, one soft-refusal row, one JSONDecodeError). Applied edit:

    + - **Judge / API-call retry wrappers treat 529 Overloaded as transient by default.** Tuple = (APIConnectionError, APITimeoutError, RateLimitError, anthropic.InternalServerError) … checkpoint/resume + per-row `judge_failed: true` audit flag instead of crashing the launcher.

  15. Target: .claude/rules/upload-policy.md — what: the env-load prefix is part of the canonical HF-resolution snippet — commit: 8ed142607 Why: #568, #520, #517, #550 (and #488's gate check) each ran the documented list_repo_files snippet without .env loaded, died on HF_TOKEN missing, then crashed again on bare load_dotenv() under a heredoc — 2-3 wasted retries per session, four sessions in one day. Applied edit:

    - `uv run python -c "from huggingface_hub import list_repo_files; …"`
+ `set -a && source .env && set +a && uv run python -c "from huggingface_hub import list_repo_files; …"`

  16. Target: .claude/skills/promote-clean-result/SKILL.md — what: execute the promote directly when the user's invocation carries explicit promote intent — commit: 44b5d8065 Why: Thomas said "Promote 488", got a status summary + re-confirmation ask instead of execution, and had to repeat "PROMOTE IT" in caps. The user-only rule guards against AUTOMATION flipping classification; an explicit human "promote N" is itself the gate (and other sessions today already executed on it). Applied edit:

    - (promotion is user-only — I cannot run it.)
+ **If the user's request already carries explicit promote intent** … run `task.py promote <N> useful` directly on their behalf. Ask ONLY when classification is ambiguous or a gate FAILed.

Other problems & notes

  • rsync is missing from the pinned pod image — 10+ independent rediscoveries today (sessions for #534, #464, #563, #556, #547, #477, #541, #533, #552, #558, #505, #548 all fell back to scp/tar-over-ssh). Suggested action: one-line apt-get install -y rsync in scripts/bootstrap_pod.sh (scripts/ = not auto-applied; route via implementer or fold into the #580 ops-hygiene batch task).
  • codex_task.py output files are reused across dispatches and not truncated at spawn — #516's stalled Codex critic left #508's content in the output file, nearly read as a current verdict. Suggested action: truncate/uniquify --output-file per job + stamp the issue number in a header (scripts/; today's 2ca659a52 added probe retries but not output isolation). The "No job found" post-spawn registry race hit ~15+ times across ≥8 sessions; verify the new retry covers the post-spawn probe path specifically.
  • task_workflow.py sharp edges: (a) _run_git has no retry on index.lock even though CLAUDE.md tells agents to retry once (a set-status crashed mid-flow on #480); (b) address-concern hard-raises on carried >200-char summaries instead of truncating. Suggested action: small implementer task (library code, not auto-applied).
  • worktree_audit.py holder detection counts transient read-only scanners (du/find/grep/tar) as holders via argv substring — during the disk crisis a stale du pinned ALL worktrees simultaneously, defeating the reaper exactly when disk-pressure mode needed it. Also: uv cache (43G) was twice the single biggest consumer and isn't touched by disk-pressure mode. Suggested action: implementer task on worktree_audit.py (treat argv-only scanner matches as non-holders; add uv cache prune to disk-pressure remediation).
  • keep-running tag never expires: pod-530 sat stopped-but-billing on a finished task until Thomas noticed and asked to terminate it. Suggested action: pod_audit/watcher alert (or auto-drop) when a keep-running pod's task is terminal with no live follow-up inferred.
  • Watcher blind spot — follow-up rounds that die before epm:run-launched: #464's user-approved follow-up passed code review then sat ~12h with no live session, no pod, and a parked-looking status until Thomas asked "did the rerun run?". Suggested action: watcher detector for "followup-scope + code-review PASS + no run-launched + no live session after ~2h".
  • Watcher respawn race: a respawned #535 session and its sleeping predecessor both drove the issue (duplicate harness run + duplicate SLURM job) before the twin was found and stopped. A fence/stop-predecessor candidate was surfaced in-session — verify it landed in autonomous_session_watch.py.
  • Preflight's 50GB fallocate probe on MooseFS costs ~18-45 min of log-silence per (re)launch — #541 paid it ≥3 times on a paid 4×H100 pod and it triggered a false "stalled" watcher verdict. Suggested action: cache the probe PASS per pod (/workspace/.preflight-disk-ok) — src/ preflight, implementer task.
  • HF caches can root INSIDE worktrees on the VM (15-16G found in issue-527/489/543/522 worktrees; one 143MB shard download killed #522's 38h sweep at its final cell). Suggested action: pin VM-side HF_HOME to a shared absolute path in orchestrate/env.py (src/, implementer task); meanwhile worktree_audit could flag */cache/huggingface dirs.
  • #519's dispatcher silently skipped phases C/D/E (optional-args-gated phases default to silent skip) — discovered 13h in; task parked as explicitly PARTIAL. Suggested action: launch-time assert that plan-enumerated phases have their args (experiment-code; follow-up on #519's line).
  • #509's fact arm silently downgraded to --smoke (production mode refused on a missing per-seed-SE reconstruction and fell back quietly) — Thomas caught it; a fix subagent was dispatched. Suggested action: verify the "planned-arm downgrades must be loud" fix actually landed.
  • Reconciler skipped on #502's round-3 Claude-PASS/Codex-FAIL split (went straight to round 4; outcome fine). Suggested action: consider a mechanical events.jsonl check (PASS+FAIL verdict pair with no reconcile marker) in workflow_lint or the critic-step prose.
  • ~/.claude/skills/humanize/check_bans.sh false-positives on quoted raw model output ("Sure, I'd be happy to help…" inside a required verbatim sample block FAILed #518's ban gate; worked around by eliding fenced blocks). User-global skill — route via /memory-sleep or fix by hand: elide fenced + <details> blocks before scanning.
  • A user-global formatter hook rewrote ~/lit-review/fetch_papers.py to a py3.11-only idiom (datetime.UTC), silently breaking the nightly digest running on system python3.10 (fixed in-session with a timezone.utc shim). Suggested action: exempt ~/lit-review/ from the py-upgrade rewrite or pin the systemd unit to uv run python — user-global hook, Thomas's call (/update-config).
  • On-stop 429-retry hook coverage: one #553 implementer subagent's terminal result was the 429 error text; whether on-stop-429-retry.sh re-fired isn't visible in the transcript. Suggested action: verify the hook matches the "Request rejected (429)" phrasing on subagent terminal results.
  • Judge-wrapper librarization: #528's hardened judge loop (checkpoint/resume + per-row judge_failed + decode retry) was rebuilt per-issue and immediately needed by #556. Fix 14 documents the pattern; the library lift into src/explore_persona_space/eval/ is an implementer task.
  • Pending decisions/loose ends for Thomas: (a) backends/** workflow-surface membership (greenlight-required candidate from the #570 session); (b) confirm the GCP A100 eps-issue-535 instance isn't idling on credits (flagged when the #535 session died of SSH hangup — the tmux auto-wrap fix landed); (c) Compute Canada robot access still blocked on fir/narval/rorqual/tamia (follow-up email sent — resurface ~Jun 12 if no reply); (d) the session-reconcile auto-stop default flipped twice on 06-10 (alert-only → auto-stop at awaiting_promotion) — surfaces are consistent with the later decision, confirm it's the standing intent.
  • Interactive-session behavior notes (auto-memory rules reinforced, no file edits): mentor-comment pastes default to inform-not-draft (Thomas: "No I will write it just give me information"); always pin the aggregation level (cell vs family) next to each ρ in mentor drafts (the −0.40 vs −0.79 catch); re-print the artifact browser link after every revision round ("where's the link" ×2); the tasks/<status>/<N> literal-path ban and uv run prefix rules each recurred ~4× despite existing rules — counters logged, no new fix.
  • Already-handled in-session (verified or high-confidence, listed for the record): Step 9b untagged-followup guard, followups_running status-hold code guard in task.py, Step 5a sync exemption for branches that legitimately edit workflow surfaces, zombie-wrapper reaper + session-reconcile auto-stop, watcher disk-headroom guard, GPU-idle advisory trio, codex-twin worktree-path fallbacks (6 improver spawns from the #550 session), methodology-top-link change, dashboard auto-refresh, #543's cuInit gotcha merge, #480 marker-recipe-overrides-parity fix, poll_pipeline CPU-bound-stall candidate (verify the CPU-time check is on main).

My thoughts

Highlighted results

  • #501 — Base-model cosine distance ranks the marker log-prob proxy across the single-turn / multi-turn format boundary, but the marker never emits on-policy so #377's behavioral silencing remains unexplained (MODERATE confidence)
  • #509 — On sycophancy the residual-stream persona-distance signal lives at early-to-mid layers across both extraction points, not at the parent #502 marker-leakage recipe's late-layer last-prompt anchor — and fact transfer underperforms #494's coarse predictors even before prior control (MODERATE confidence)
  • #549 — Auditing every git-tracked vLLM-LoRA eval driver for the cache-collision bug finds one new affected published result — #504's saturated-anchor geometry was measured on the step-6 adapter, not step 25 — and the bug is still live on main (HIGH confidence)