daily-fix: CVD-scoped nvidia-smi + schema-from-artifact
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Workflow gap
Two expensive 2026-08-05/06 incidents each recur a documented class that no reviewer or implementer checklist currently catches:
- nvidia-smi verdicts not CUDA_VISIBLE_DEVICES-scoped (re-introduced #1333 class).
#2091's
issue2091_pod.py::reap_generation_enginetookmax()ofmemory.usedover ALL 4 GPUs (nvidia-smi ignores CVD), so 4 of 9 rung-jobs whose own GPU read 0 MiB died: "RuntimeError: vLLM teardown did not drain below 2048 MiB within 180s (per-GPU used MiB: [(0, 35579), (1, 19143), (2, 0), (3, 0)])" — ~765–880 s lost per job + a fix round (commit 2cc130dbff scoped the verdict own-device). The gotcha is documented; the review surface never checks for it. - Loader schema fabricated from memory instead of read from the artifact. #2061's
round-1 implementation fabricated the #1336 shard schema — review verdict: "the
pipeline cannot load its own input (fabricated
#1336shard schema), P2 crashes deterministically after the expensive fits" — caught pre-GPU by the adversarial loop but at the cost of a full implementation round (~4.5 h wall) + review round 1 of 5. Related same-day sibling: #2091's judge collector assumed every packed row is a rollout and KeyError'd on the packed_manifest.jsonrow 0.
verified-at-filing: both incidents are probed marker/review readbacks (miner 6 P4 rows
1088/1131 + fix commit; miner 5 P6 rows 108–118). grep -cn 'nvidia-smi' .claude/rules/code-reviewer-section-reference.md .claude/agents/experiment-implementer.md and grep -cn 'schema' .claude/agents/experiment-implementer.md
run at compose time to confirm neither surface carries the item.
Proposed change
.claude/agents/code-reviewer.md(or its section-reference file): checklist item — any nvidia-smi-based drain/teardown/free-memory verdict in fan-out or dispatcher code must be scoped to the job's own CUDA_VISIBLE_DEVICES row(s); a whole-host max/min is a FAIL..claude/agents/experiment-implementer.md: before writing any loader for a banked artifact, download and open ONE real shard/sidecar and paste its observed keys into the implementation marker (schema-from-artifact, never schema-from-memory); packed-format consumers filter rows on the pack'ssrc/schema field.
Provenance
-
fingerprint: aab30f0c53a8
-
workflow_fix_target: .claude/agents/code-reviewer.md, .claude/agents/experiment-implementer.md
-
origin: /daily 2026-08-05 problem sweep — miner 6 P4/P6, miner 5 P6.