EPS
← All docs·Activity

Daily — 2026-06-27 (#664 vLLM-crash saga;

updated: 2026-06-28

What happened

  • #664 vLLM EngineCore-crash saga dominated the day: ~8 GPU launches / 4 strategy pivots against a reproducible hang inside vLLM 0.11.0's first llm.generate() on the _elicit_secure_code path (across pods, batch sizes, eager mode, V1/V0 engine). The saga drove several merged workflow-fixes — poll_pipeline.py zombie-GPU-allocation detection, gotchas.md vLLM chunk-by-default + pkill-reaper recipes, failure_classifier.py + SKILL.md routing the zombie-GPU stall_reason to infra. Final unexplored lever (enable_prefix_caching) parked.
  • #658 leakage-theory experiment authored its clean-result (epm:analysis v1): geometric context distance does NOT predict leakage; the apparent 3/10 Betley PASSes were multiple-comparison artifacts beaten by a random-projection control (0/10 after the attn recipe + control landed). Not yet promoted.
  • 9 workflow-fix / infra tasks completed: #686 (GcpBackend confirm-deleted teardown), #687 (verify_task_body.py figure-text-staleness), #688 (GCP janitor whole-project sweep), #692 (watcher RunPod no-port wedge), #693 (parallelize #664 p0 8-way), #694 (results-driven literature-positioning gate), #695 (idle-unmapped tmux reaper), #674 (per-cell .npz → local scratch), #675 (_context_vector_all_layers via hook helper).
  • Late-night theory + setup chat (sessions a1ea2387 / f721a47c / d773b1fe / 72c44523 / 8a7e6899): Δ_{C,B} vs B=B′ leakage discussion, #532 marker-change lookup, launched marker+sycophancy behavior-dependent-key test in background, added concise/verbose/casual register behaviors + commit-push to #660, asked for (and got) #684 the $400 throughput-validation run.
  • workflow-improver agent deprecation (#678) landed mid-day but several already-running autonomous sessions (#664, #653, #683) still spawned the retired agent before the propagation — the deprecation banner + lint check are now in place.

Applied workflow improvements

  1. Target: .claude/skills/issue/SKILL.md (Step 4a) — what: worktree-safe REPO_ROOT derivation — commit: d56a9bdc0f Why: A resumed/crash-respawned /issue session whose cwd was inside the worktree made REPO_ROOT=$(git rev-parse --show-toplevel) return the worktree path, so WORKTREE="$REPO_ROOT/.claude/worktrees/issue-N" doubled to .../issue-N/.claude/worktrees/issue-N — every guard cd'ing to it exit-128'd (fatal: cannot change to ...), silently defeating the Step-10d foreign-tasks merge guard. Seen across ~28 sessions (session 962f1103 #690: REPO_ROOT=...worktrees/issue-690 WT=...worktrees/issue-690/.claude/worktrees/issue-690). The Step-10d guard already uses the robust idiom (line 6266); Step 4a did not. Verified empirically: --git-common-dir resolves the true repo root from both the repo root and a worktree; --show-toplevel returns the worktree path. Applied edit:

    -REPO_ROOT=$(git rev-parse --show-toplevel)
+# Worktree-safe REPO_ROOT: --git-common-dir resolves the MAIN repo root even
+# when this snippet runs with cwd INSIDE a worktree (a resumed / crash-respawned
+# session). [...]
+REPO_ROOT=$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")
 WORKTREE="$REPO_ROOT/.claude/worktrees/issue-<N>"

  2. Target: .claude/skills/adversarial-planner/SKILL.md (Phase 1.5.0) — what: document the verify_plan.py --json consumer (exit code + overall key) — commit: 052416f9d7 Why: The orchestrator improvised data['verdict'] / data['overall_status'] against verify_plan.py --json 3 times today and KeyErrored (sessions a633d835 #692, 962f1103 #690). The payload has no such keys — the verdict field is overall ("PASS"/"FAIL") and the exit code (0=PASS/1=FAIL) is the canonical signal. Verified against _json_payload in verify_plan.py (keys: source/issue/kind/overall/n_fail/n_warn/n_skip/checks). Also warned against 2>&1 poisoning json.load(sys.stdin) with the uv VIRTUAL_ENV stderr warning. Applied edit:

    +- **Reading the verdict — use the EXIT CODE, not a parsed key.** verify_plan.py
+  exits 0 on PASS and 1 on FAIL [...] the verdict field is `overall` [...]
+  there is NO `verdict` / `overall_status` top-level key [...]

  3. Target: scripts/pull_results.py — what: fix wandb ArtifactCollection.versions() → artifacts() — commit: ac7b0f719b Why: wandb ≥0.16 renamed ArtifactCollection.versions() to artifacts(); the old call raised AttributeError, caught by the bare except and silently masked as "No eval-results artifacts found" (session 6c469aa1 #685). Verified on installed wandb 0.25.0: ArtifactCollection has artifacts, not versions; inner Artifact attrs (version/created_at/size/metadata/name) all survive the swap. Syntax + API-resolution + ruff all pass. Applied edit:

             for collection in collections:
+            # wandb >=0.16 renamed ArtifactCollection.versions() -> artifacts(); [...]
-            for version in collection.versions():
+            for version in collection.artifacts():

  4. Target: .claude/agents/codex-clean-result-critic.md (Lens 12) — what: ≤30-word / ≤60-word caps are WARN, not gate-FAIL — commit: ba266c25d6 Why: The Codex twin FAILed a 31-word Takeaways bullet (1 word over) that verify_task_body.py check 20 only WARNs over and the Claude critic correctly PASSed (session 84817572 #683) — escalating a WARN-level cap to a blocking FAIL drove needless revise rounds. Aligned Lens 12's verdict to the verifier's WARN boundary. Applied edit:

    -- Takeaways bullets ≤30 words, figure captions ≤60 words (verifier
-  check 20 WARNs over both): PASS|FAIL
+- Takeaways bullets ≤30 words, figure captions ≤60 words: these are
+  **WARN-level** caps [...] do NOT FAIL the lens on a 1-3-word cap overage: WARN

Other problems & notes

Filed as a task (too-big-for-same-turn, needs full pipeline):

  • events.jsonl post_event non-atomic append corruption (session b8d3dbc3 #653) — a SIGKILL'd watcher left a 129-byte partial JSON line that crashed every task.py view/list_events/dashboard read and stalled /issue 653 resume >2h. Filed #699 (workflow-fix: atomic post_event append) + auto-spawned — fixing the core task-state append path needs a concurrency/crash regression test + full review, not a midnight quick-patch. Target: src/explore_persona_space/task_workflow.py.

Held — judgment calls (carve-out in parens):

  • #658 MLP LOCO fit ran on CPU, needlessly GPU-starved (_fit_mlp_loco, 300-epoch AdamW per cell; session f721a47c) — Thomas flagged "why are we running this on CPU?". Routing a torch-MLP fit to GPU is a real fix but the planner's "CPU-only analysis phases" rule deliberately keeps cheap stats off pods; whether this specific fit is GPU-worthy is a scope call (scientific-meaning). Suggested: add a planner/critic note that a torch-MLP LOCO fit is NOT a cheap CPU stats step.
  • #664 OOM crash-loop relaunched with identical strategy (r4_..._OOM at accelerate _convert_to_fp32, 57GiB; session bfd0775d) and chunk-500/50 relaunched on a falsified hypothesis (no [vllm-chunk] log line meant the hang preceded the first chunk; session 2c432067) — whether to skip the offending cell vs fix the fp32-accumulation extractor is an experiment-scope decision (scientific-meaning). Suggested: experiment-implementer.md note — confirm a fix's code path is reached (log line) before re-running.
  • #664 ~8-launch saga burned substantial GPU-hours on an undiagnosed hang (sessions 2c432067/b3489bdb) — the cost is the lesson (spends-money). Suggested: a hang at vLLM generate() should trigger py-spy/enforce_eager/prefix_caching differential diagnosis on ONE pod before reprovisioning a fresh 8×H100.
  • #653 spend-approval gate (132 GPU-h > 100h cap) silently un-parked on resume because the respawned session's env carried EPM_PLAN_AUTOAPPROVE_GPU_HOURS=150 (session 62c6451a) — a once-parked spend gate auto-approving on a cap bump is a spends-money path; making the park sticky touches the plan-gate contract. Suggested: a user-facing spend park should require an explicit user-approval marker, not a config-drift cap.
  • #664 Mistral-Small-3.1-24B tokenizer regex warning on every eval cell (session 09e41486) — "incorrect tokenization" warning on a Qwen-2.5-7B project suggests a mis-tagged merge; setting fix_mistral_regex=True vs verifying the base tokenizer is a results-integrity call (scientific-meaning).
  • #658 near-premature "complete" before the random-projection control arm landed (the 3/10 PASS was an artifact; session 3149da48) — caught correctly, but the analyzer ran on a partial grid. Suggested (scientific-meaning): analyzer.md guard — confirm all planned control arms present before authoring verdicts.

Held — already fixed / filed in-session today (no action; noting so nothing looks dropped):

  • GCP FLEX_START zombie (EXIT-trap DELETE no-op) → #686 completed. verify_task_body.py figure-at-SHA / sidecar resolution → #687 completed. GCP janitor name-filter blind spot (eps-cap-probe2 ~$14) → #688 completed. #664 zombie-GPU detection / chunk recipe / stall_reason routing → merged in-session. #693/#694/#695/#674/#675 → completed.
  • workflow-improver spawned by in-flight sessions (#664/#653/#683) after the same-day #678 deprecation — banner + lint check now in place; sessions predated propagation.

Held — can't verify tonight (logged, not committed):

  • Full-suite test-isolation pollution: test_autonomous_session_watch.py::test_vm_reclaim_hf_hub_cache_evicts_through_bounded_worker, two test_issue642_analyze_decision_rule tests, and test_issue672_validation.py::test_loop_poller_detects_second_failover_in_quiet_period FAIL under full-suite ordering, PASS in isolation (global-state leak). Erodes the Step 9c gate (which special-cases them). Diagnosing the leaking global + confirming a full green suite is real work I can't verify cheaply tonight. Suggested: file a kind: infra task to add autouse teardown fixtures resetting the polluted module-level singleton.
  • last_log_mtime_sec_ago clock/epoch bug in backend_poll.py (grew to 14–33h on a log with ~30s mtime; session 09e41486) — masked only by the CPU-advancing override; would falsely trip the >900s stall path if that override were removed. Needs reproduction against the pod clock to pin the epoch/timezone skew.
  • confirm_artifacts finalize gate fails on phase-scoped / multi-attempt launches (forces --skip-confirm-artifacts every time; sessions 92450661 #661, 6c469aa1 #685) — resolves the repo-root/main tree + stale attempt sentinel instead of the issue-branch worktree + current attempt. Real scripts/dispatch_issue.py/backends bug; needs careful finalize-path verification. Suggested: file kind: infra.

Held — orchestrator process-mistakes despite correct docs (discipline, not a doc gap):

  • Step-10d recovery improvised hand-built git rm tasks/approved/675 and cat-file origin/main:tasks/reviewing/674 probes (sessions bb4dc928/9dd9e79f) — SKILL.md's post-merge guard already mandates task.py find + ls-tree | grep (no hand-built paths, no cat-file fatal probes). The orchestrator deviated.
  • task.py post-marker --note … --file … mutual-exclusion error hit ~4–6×/day (multiple sessions) — CLAUDE.md already documents --file for long bodies; agents still pass both. A code-side change (auto-prefer --file) would help but is a CLI-contract change (held: public API contract).
  • 9b step-id: SKILL.md prose narrates "Step 9b" but workflow.yaml § steps only registers 9b-same, so post_step_completed.py <N> 9b errored (session 6c469aa1 #685, non-fatal). Ambiguous which is canonical (add 9b vs fix the caller vs don't post) — held (ambiguous-intent / step-enum contract).
  • Foreground sleep && tail polling blocked by the harness guard (multiple sessions); bare python3 -c / … | python3 -c used instead of uv run python (works only because python3 happens to exist). Both self-recover; reinforced rules exist.
  • Stale VIRTUAL_ENV=/home/thomasjiralerspong/introsp/.venv prints a uv warning on nearly every uv run (~130 session files) — benign (uv ignores it); the fix location is the Happy daemon's inherited env, not spawn_session.py (which POSTs to the daemon, doesn't build a subprocess env). Suggested: unset VIRTUAL_ENV in the daemon launch, out of scope for a code patch here.

My thoughts

Highlighted results

  • no results promoted today