Make pre-teardown verification sweep the out-root TOP LEVEL: three artifact losses on one task, each caught only by a manual name-set diff
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Goal
Close a demonstrated, repeating durability gap: per-issue phase_upload
implementations glob their SUBDIRECTORIES and therefore silently omit files
written at the out-root TOP LEVEL. Each such file is absent from HF, absent
from git, and not a declared discard — so it dies at pod teardown, and the
only thing standing between it and permanent loss is an agent remembering to
run a manual sweep.
Task #2162 hit this three separate times in one run. That is a class, not three coincidences.
The three incidents (all #2162, all confirmed)
- P5 —
pilot_gate_report.json(740 B). Sat at/workspace/issue2162_out/top level;phase_uploadglobs only the six subdirs. Caught by theupload-verifier, which FAILed the run on exactly this. It recorded the MEASURED per-rollout generation cost (1.2019 s/rollout at gen_batch=16 over 8 workers) that the poll fence and ETA were derived from — a successor would have had to re-run the pilot to recover it. - P8 —
stage2_results.json(754 B). Same shape: out-root top level, whilephase_uploadglobs onlyshard_*.jsonlinsidestage2/. Carried the stage-2 reproducibility block plus two scope notes, one of which states that the plan'sphase_outputs.P8V_a-shard entry is DELIBERATELY not produced — without it, a later upload-verify reconciling P8 against the plan would FAIL hunting an artifact that was never meant to exist. - Margin leg —
upload_done.json(2,580 B). The manifests glob is['*.json', 'blocks/*.done.json', 'margin_blocks/*.done.json']andupload_done.jsonis written AFTER that upload completes, so the upload-completion marker structurally cannot be inside its own upload (236 files on the pod vs 235 uploaded). It carried the fullreproducibility_card(seed_base, bank_seed, max_new_tokens, temperatures, draw counts, gen_batch, library versions) AND three plan deviations recorded nowhere else, all of which the report Methodology needs.
Incident 3 is the sharpest: the loss was found only because a recursive pod-vs-HF name diff was run by hand before teardown. A count-only check would have read 235 uploaded and looked fine.
Why the existing safety nets did not cover it
- The
upload-verifierDID catch incident 1 — so the workflow surface works when it runs. But it runs at Step 8 / P5-style checkpoints, not before every suffixed-pod or deferred-leg teardown, which is where incidents 2 and 3 lived. - The completion-side teardown contract says "verify THIS round's artifacts, then terminate", but "verify" is not currently specified to include an out-root top-level sweep, so a conscientious agent can verify the declared prefixes 100% and still lose a file.
- Producer self-reports are useless here by construction: each upload logged "Bulk upload verified: N files" and each N was CORRECT for what it globbed. The omission is in the glob, not the upload.
Candidate work (the implementing session decides the shape)
- Primary — make the sweep a specified part of pre-teardown verification.
Amend the completion-side teardown recipe (
.claude/rules/pods.mdand the CLAUDE.md "Completion-side teardown" clause) so verify-then-terminate explicitly includes: enumerate every file the run produced under the out-root (recursively), diff NAME SETS against the union of uploaded prefixes, and either upload or git-commit any residue before terminating. Name the count-only trap: a matching count is not a matching set. - Consider a mechanical check in
upload-verifier(and/or a helper the per-issue upload code can call) that fails loud when any regular file under the out-root matches NO upload glob and is not indiscarded_artifacts. This is the version that actually prevents a fourth occurrence, because it does not depend on agent diligence. - Consider the chicken-and-egg case explicitly. An upload-completion marker written after its own upload can never be inside it. Either write such markers BEFORE the final upload, or give them a dedicated second tiny upload, or route them to git. Whichever is chosen, state it so per-issue authors stop reinventing it.
- Consider
.claude/rules/upload-policy.mdguidance for per-issuephase_uploadauthors: glob the out-root top level, not only subdirs.
Scope notes
- The offending globs live in per-issue experiment code
(
scripts/issue2162_run.py,scripts/issue2162_stage2.py), which is NOT workflow surface and should not be retro-edited for a completed run. The durable fix belongs in the RULES and the shared verifier, so the next experiment's author inherits it. - Do NOT weaken the existing fail-loud posture: a genuine discard still
requires the declared
discarded_artifactsentry with a regen recipe, and text/JSON is never discardable. - All three #2162 losses were recovered before teardown, so nothing is
currently missing — this task is about preventing the fourth, not repairing
the first three. The recovered files are committed at
9fddd9e1a6,6eec51c77e, and92f25415eerespectively if the implementer wants to see the payload shapes.
Provenance
- workflow_fix_target:
.claude/rules/pods.md - Also amends:
CLAUDE.md,.claude/rules/upload-policy.md,.claude/agents/upload-verifier.md(+ its section reference), and the sharedscripts/verify_uploads.py. - Filed by the #2162 orchestrator via the workflow-fix-on-bug protocol after the third out-root top-level artifact loss in one run.
- This session runs under a
workflow_fix_target:Provenance line — it MUST NOT auto-route its own subagents' workflow-fix candidates (recursion guard).