Size-gate the CPU-off-pod rule: route large-data CPU-only phases to a CPU-only GCP instance (gpu_count=0 n2-highmem) instead of the shared VM
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Problem
Large-data CPU-only analysis phases run on the shared CPU orchestration VM
per the "CPU-only phases run off-pod on the VM against uploaded artifacts" rule
(planner.md §9, CLAUDE.md "CPU-only phases don't hold GPU pods"). That rule was
written for SMALL phases (bootstrap/permutation stats, judge scoring, eval-JSON
aggregation). It has no size guard — so #658's predictor-fitting
(issue658_fit_predictors.py) materialized a 139 GiB activation store in its
worktree on the 485 GiB shared VM and ran for 15h+, filling root disk to 96% and
choking foreground Bash spawns across all VM sessions (incident 2026-06-26).
Change
Make the CPU-off-pod rule size-gated:
- SMALL CPU phases (working set below threshold) → stay on the VM (unchanged).
- LARGE-DATA CPU phases (working set ≳ threshold, default ~30–50 GiB) → run on a CPU-ONLY pod/instance that downloads its inputs from HF, runs, uploads results, and tears itself down — decoupling big stores from the shared VM.
Feasibility (VERIFIED 2026-06-26, before filing)
- GCP CPU instance = clean primary path. GCP accelerator-optimized machine
types BUNDLE the GPU into the machine type (gcp.py: "the a2-ultragpu-* family
hardcodes the accelerator; not threaded into gcloud") — so there is NO
--acceleratorflag to gate; a CPU-only instance is simply a CPU machine type (e.g.n2-highmem-16) with no GPU.backends/gcp.get_machine_spec's override path already acceptsgpu_count=0. GCP CPU capacity is abundant (theZONE_RESOURCE_POOL_EXHAUSTEDstockout is GPU-only), so this lane is reliable even while A100s are dry. - RunPod CPU pod = possible but net-new code. RunPod offers CPU pods, but
scripts/runpod_api.create_podhard-assertsgpu_count >= 1(≈line 501) and uses the GPU-onlygpuTypeIdcreate mutation — a CPU pod needs RunPod's separate CPU-deploy API path. Implement as an OPTIONAL later fallback, not now.
Implementation (GCP-CPU lane)
- New CPU intent (e.g.
cpu-bigmem) →MachineSpec(machine_type="n2-highmem-16" or sized to the working set, gpu_count=0, gpu_kind="CPU")with a large--boot-disk-size(≥ store size + headroom). Add toINTENT_TO_MACHINE(or the override path) +pod.pyintent table +--list-intents. gcp.pycreate handlesgpu_count=0: CPU machine type;--maintenance-policy=MIGRATE(CPU VMs can live-migrate, unlike GPU which must TERMINATE); SKIP the accelerator-quota preflight whengpu_count=0(no accelerator quota — gate on CPU/disk quota instead); optionally a non-CUDA image (DLVM works on CPU but is wasteful). Keep--instance-termination-action=DELETE+--max-run-durationso it stays ephemeral.- Router /
dispatch_issue.py: expose the CPU intent; the large-data CPU phase on the instance downloads the store from HF (huggingface_hub), runs, uploads results to HF / commits eval JSON, then the instance auto-deletes. (No GPU pod held; no 139 GiB on the shared VM.) - Rule update (size-gated):
planner.md §9+ the CLAUDE.md "CPU-only phases" bullet — small → VM, large (≳ threshold) → CPU GCP instance;critic.mdMethodology lens item 10 enforces it. Name the threshold + how the planner estimates the working set. - Tests:
tests/test_gcp_backend.py—gpu_count=0renders a valid CPUgcloud instances createargv (CPU machine, no accelerator,MIGRATE, big--boot-disk-size, no accelerator-quota call); a routing test that a large-working-set CPU phase selects the CPU intent and a small one stays on VM.
Scope / acceptance
- A CPU-only GCP instance can be provisioned (
gpu_count=0, CPU machine, big disk) and run a CPU analysis phase that pulls its store from HF — verified at least by argv-render golden tests; a live smoke (provision a realn2-highmem, pull a small store, run, delete) is a nice-to-have if cheap. - Large-data CPU phases route off the shared VM; small ones stay (size-gated).
backends/*.pyis workflow surface — full /issue treatment (planner/critic/code-review).- RunPod-CPU path is explicitly OUT of scope here (optional later fallback).
Provenance
User directive 2026-06-26 ("if there's large data involved, run it on a CPU-only RunPod or GCP pod") after the #658 139 GiB-store-on-the-shared-VM disk-fill incident. Feasibility verified before filing (GCP-CPU clean; RunPod-CPU needs new create path).