EPS
← All tasks·#2597Proposed

representation_shift teardown: guard torch.cuda.ipc_collect() on is_initialized() — unguarded call lazy-inits CUDA at teardown

kind: infra
track:

Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.

No comments yet.

Goal

Guard the torch.cuda.ipc_collect() call at src/explore_persona_space/analysis/representation_shift.py:484 behind torch.cuda.is_initialized(), so the documented vLLM-teardown recipe cannot CREATE a CUDA context while trying to release one — and so it stops crashing CUDA-less hosts.

Evidence — verified against installed torch source, not inferred

torch 2.8.0+cu128
ipc_collect   _lazy_init=True   is_initialized_guard=False   ->  _lazy_init()
empty_cache   _lazy_init=False  is_initialized_guard=True    ->  if is_initialized():

torch.cuda.ipc_collect() calls _lazy_init() unconditionally. torch.cuda.empty_cache() is internally guarded. The asymmetry is the whole issue: the two are called as a pair, and only one of them is safe on a CUDA-untouched process.

The call site (representation_shift.py:478-486) calls both unguarded:

_reap_vllm_engine(llm)
del llm
gc.collect()
torch.cuda.empty_cache()
torch.cuda.ipc_collect()  # complement to empty_cache for inter-process freed mem
time.sleep(1.0)

Why it matters

This block is the reference implementation that .claude/rules/gotchas.md § vLLM teardown points every caller at ("Reference impl: representation_shift.py::_reap_vllm_engine"), so the defect propagates by citation. Two concrete failure modes:

  1. A CUDA-untouched parent process. Under vLLM v1 the EngineCore runs in a separate subprocess, so a spawn-mode parent can drive an engine without ever initializing CUDA in its own process. Running the teardown recipe there makes ipc_collect() initialize CUDA at teardown — allocating a fresh context at the exact moment the code is trying to free resources.
  2. CUDA-less hosts. Any test host or CPU-only box executing this teardown path raises from _lazy_init() rather than no-opping, which is the opposite of what a cleanup helper should do.

Gating empty_cache() too is harmless but redundant (it self-guards); ipc_collect() is the load-bearing fix.

Provenance — found by adjudicating a deliberate deviation

Surfaced during /issue 2546's post-PASS code review. The #2546 round-12 implementer, wiring this same recipe into scripts/issue2546_gen_capture.py's gen-worker terminal, deliberately DEVIATED from the reference by gating both calls on torch.cuda.is_initialized(). The orchestrator flagged that deviation to reviewers as a genuine fork requiring a decision, not a style call: either the deviation is correct and the reference has a latent trap, or the gate can skip a needed free.

The Claude reviewer adjudicated it from torch source and concluded the deviation is CORRECT and the unguarded reference is the latent trap, recording it as a standing recommendation out of that round's scope. The orchestrator then re-verified the torch behavior independently (output above) before filing.

So the in-repo copy at #2546 is already correct; this task fixes the SHARED reference the rule tells everyone else to copy.

Scope

  • Guard ipc_collect() (and, for symmetry and explicitness, empty_cache()) on torch.cuda.is_initialized() at representation_shift.py:484.
  • Sweep for other unguarded ipc_collect() calls repo-wide and apply the same guard; report the enumeration.
  • Add a regression test that executes the teardown block on a CUDA-untouched process and asserts CUDA is still uninitialized afterward (torch.cuda.is_initialized() False) — the property that actually distinguishes fixed from broken, and which is checkable without a GPU.
  • Update .claude/rules/gotchas.md § vLLM teardown where it spells out the recipe (... + ipc_collect()+time.sleep(1.0)``) to state the is_initialized() guard, so future copiers inherit the fix rather than the trap.

Do not change the reap ORDER or drop any step — the sequence is load-bearing per the rule (engine-core shutdown, then destroy_process_group, then del/gc/cache/ipc/sleep). This is a guard addition only.