workflow-fix: gotchas.md — manual launch of dispatcher-managed workers must replicate launcher CVD env pin
kind: infra#wf-fix#wf-fix-fp:cda669c3bafa
track:Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview / Motivation
Auto-filed by the workflow-fix-on-bug protocol from the orchestrator's own observation on task #813 (supervisor intervention, 2026-07-02 16:51Z).
Goal
Extend the gotchas.md CVD entry so a manual/supervisor launch of a dispatcher-managed per-cell worker replicates the dispatcher's launcher-env pins (CUDA_VISIBLE_DEVICES et al.) instead of relying on --gpu-id.
Workflow gap
- Bug observed: supervisor hand-launched 4 issue813_run_cell.py workers with only
--gpu-idand no CUDA_VISIBLE_DEVICES pin; all 4 vLLM engines initialized on the busy default GPU and crashed at init_device (~4 min lost; locks auto-released; relaunch withenv CUDA_VISIBLE_DEVICES=<g>succeeded). - Why it is a workflow gap: .claude/rules/gotchas.md documents the
+gpu_id=NCVD clobber for Hydra launches, but not the operational rule that per-cell worker scripts in this codebase treat--gpu-idas informational (_devicereturns cuda:0) and depend ENTIRELY on the launcher env pin — the dispatcher does it (issue813_dispatch.py_build_cell_cmd, the #545 pin), so any HAND launch that copies only the CLI args reproduces this crash. - Confidence (emitter): high
Proposed change (candidate diff sketch — refine in planning)
In .claude/rules/gotchas.md, extend the CUDA_VISIBLE_DEVICES clobber entry:
- Hand-launching a dispatcher-managed per-cell worker (issue813_run_cell.py class —
- any script whose --gpu-id help says "CVD-pinned by the launcher") MUST replicate
- the dispatcher's launcher-env pins:
env CUDA_VISIBLE_DEVICES=<g> ... --gpu-id <g>. --gpu-idalone is informational (_devicereturns cuda:0); without the env pin- every engine initializes on the busy default GPU and dies at init_device
- (#813 supervisor relaunch, 2026-07-02). Read the dispatcher's _build_cell_cmd/env
- before composing any manual launch.
Scope / surfaces
- Primary target:
.claude/rules/gotchas.md - Grep before editing:
grep -rln 'CVD-pinned by the launcher' scripts/to name the affected worker-script class in the rule.
Constraints / invariants
- Workflow-surface only. ruff/lint N/A (markdown). LESSONS.md index row unchanged (gotchas already indexed).
Provenance
- workflow_fix_target: .claude/rules/gotchas.md
- fingerprint: cda669c3bafa
Activity