Hoist a process-safe atomic-write helper into shared code and sweep the shared-tmp pattern out of scripts/
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Goal
Replace the process-UNSAFE tmp = path.parent / (path.name + ".tmp") atomic-write
pattern, currently duplicated across ~50 scripts/issue*_*.py files, with a single
shared process-safe helper, and migrate the call sites.
Why (proven, not hypothetical)
Task #2329 lost a pre-launch smoke round to exactly this bug. Its 8-worker grid
phase crashed on real weights:
File "scripts/issue2329_run.py", line 1258, in _write_json_atomic
os.replace(tmp, path)
FileNotFoundError: [Errno 2] No such file or directory:
'.../manifests/pe_exclusions.json.tmp' -> '.../manifests/pe_exclusions.json'
Mechanism: when N processes write the SAME destination path, they all derive the
SAME temp name. The first worker's os.replace(tmp, path) consumes the shared
temp; every other worker then calls os.replace on a temp that no longer exists
and dies. A pre-fix probe measured this as deterministic, not a rare race —
5/5 rounds failed, 7/8 workers each round.
Note the content was NOT the problem in #2329: every worker computed byte-identical content, so the "concurrent same-content writes are safe" design was sound. Only the temp-path derivation was wrong. A sweep should preserve that design rather than re-architecting call sites toward locking or single-writer election.
Why this is a latent trap rather than a dormant curiosity
Every one of the ~50 sibling scripts is single-process TODAY, so nothing is currently broken. But this project's standing rule is to vectorize / parallelize / shard by default, and per-GPU process fan-out is the normal shape for grid and sweep phases. Any script that gains a multi-worker phase writing one shared manifest inherits this crash — on a PAID run, since the failure needs real multi-process execution and no static check can reach it.
The fix pattern (already proven in #2329, commit 27206c15d9)
A contextmanager that:
- derives the temp as
f"{path.name}.{os.getpid()}.{uuid.uuid4().hex[:8]}.tmp"— pid alone is insufficient (pid reuse), the uuid component is load-bearing; - keeps the temp in the DESTINATION'S OWN directory, never
/tmp— a cross-filesystemos.replaceis not atomic; os.replaceon success;unlink(missing_ok=True)then RE-RAISE the original exception on failure, so no orphan*.tmpresidue is stranded (out-root residue is separately gated by the upload-verifier) and fail-fast discipline is kept.
Scope
- Land the helper in shared code (e.g. under
src/explore_persona_space/) so scripts import rather than re-implement it. - Sweep
scripts/for thename + ".tmp"derivation and migrate call sites, preserving existing signatures and idempotent semantics. - Add a lint check so the unsafe derivation cannot reappear.
- Pin with a real N>=8 PROCESS concurrency test on one shared destination (it must fail against the unsafe derivation) plus a no-orphan-residue assertion.
Notes
- Filed from #2329's run-phase crash-fix round; the implementer's
epm:new-bug-classtag for it isshared_tmp_name_concurrent_atomic_write. - Left at
proposeddeliberately — capture is not dispatch. Nothing is broken today, so this wants triage rather than an immediate autonomous session. scripts/issue2329_analysis.pywas named as one of the affected siblings.