workflow-fix: Step 9c pristine-compare falsely attributes pre-existing main reds to the branch (traceback-flag asymmetry + base_identical not consulted for ordinary nodes)
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Goal
Stop the Step 9c pristine-compare from attributing PRE-EXISTING main-side reds to the branch
under review. Two independent mechanisms do this today, and #2537's gate hit BOTH in one run —
every one of its new entries was false. Defect A is a traceback-flag asymmetry in the scan-node
offender-path diff. Defect B is that the base_identical sibling-sync suppression, which exists
precisely for this class, is never consulted for ORDINARY (non-scan) test nodes.
Measured on #2537: compare --run-pristine returned new: 2, stripped: 3,
indeterminate: false. Both new entries were false — one per defect — while all three strips
were correct. So the instrument's false-positive rate on that run was 2 of 5 reds, and a session
trusting it would have bounced a clean round.
Evidence
Found during #2537's Step 9c gate (2026-08-24). Reproduced end to end against that round's own committed gate junit.
The running pristine replay emitted:
SCAN-NEW-VIOLATION: tests/test_shared_vm_thread_caps.py::test_no_new_torch_before_dotenv_vm_entrypoints
— branch adds violation path(s) absent on pristine main: ['tests/test_shared_vm_thread_caps.py']
In _resolve_scan_violation_setdiff (scripts/step9c_baseline.py:3105) a non-empty branch-only
path set routes the node into ctx.new, which yields rc=1 — i.e. "the branch introduced a new
violation". It did not.
Root cause. extract_violation_paths (scripts/step9c_baseline.py:403) matches rows with
^[ \t]*(?:E[ \t]+)?(?:[-*][ \t]+)?((?:scripts|src|tests|configs|docs)/[\w./-]+\.[A-Za-z0-9_]+)
anchored at line start, tolerating the pytest E gutter, and admitting tests/ paths. The two
sides carry different text because they run under different flags:
- branch side — the Step 9c gate runs
-v --tb=short, so the failure element text contains a traceback header line,tests/test_shared_vm_thread_caps.py:1000: in test_no_new_torch_before_dotenv_vm_entrypoints; - pristine side — compare's own per-file commands run
-q --tb=no, so that line is structurally absent.
A traceback header begins with a tracked path, so the anchored regex extracts it as an offender.
v_branch then holds a path v_pristine cannot contain, and the set difference names the test
file itself as a branch-only violation.
Decisive reproduction, real extractor against the real gate artifact
(/tmp/step9c-junit-issue-2537.xml, the #2537 gate junit):
| extraction site | paths extracted |
|---|---|
junit failure message attribute | 1 — scripts/issue2254_firstk_ctxext_sensitivity.py |
| junit failure element text | 2 — that offender plus tests/test_shared_vm_thread_caps.py |
The message attribute is clean; only the element text carries the phantom.
Why the existing guards miss it
- The #2319 anchoring fix removed the quoted-snippet and remediation-prose classes (a later token on a row is no longer scanned). It cannot remove this class: a traceback header genuinely begins with a tracked path, so under an anchor-only rule it is indistinguishable from an offender row.
- The
assert-leading strip does not apply — the header does not start withassert. - The
...elision filter does not apply — no elision marker present. - The
ctx.base_identicalsuppression (#2302/#2296) does not fire. That set is the diffed-but-identical Step 5a sibling-sync class, and the flagged file is not in the branch diff at all, so it never enters the suppression set — even though it is byte-identical toorigin/main(sha30b19ba74a725898on both sides). That gap is worth deciding on deliberately: "byte-identical to main" is the property that should defeat a branch attribution, and today only the narrower "in the diff and identical" does.
Defect B — base_identical suppression missing for ordinary nodes
Same run, different node: tests/test_workflow_yaml.py::test_gates_full_shape was reported
new (branch-attributed). It is red on current origin/main, proven statically:
b116c2e872(2026-08-24 13:12Z) added theclarify_experiment_askgate to.claude/workflow.yaml,CLAUDE.md, and.claude/skills/clarify/SKILL.md— but did NOT updatetests/test_workflow_yaml.py, whoseconditional_namesassertion pins SEVEN names. Main's yaml now yields EIGHT. The assertion cannot pass on main. (Filed separately as #2566.)- The pristine oracle is the branch's MERGE-BASE — verified:
merge-base(origin/main, issue-2537)== the oracle checkoute7a80a12db, whichorigin/mainis 824 commits ahead of, and which predatesb116c2e872. At that base the gate does not exist, so the test PASSES. - #2537's branch carries the newer yaml only because the Step 5a spec-freshness sync imported
it: the branch copy is byte-identical to
origin/main's (sha 3ae66a74aee9e7a3both sides), andtests/test_workflow_yaml.pyis byte-identical too (5bc9336f2845efa6).
So the branch holds main's OWN content, the oracle predates it, and the delta is attributed to the
branch. That is exactly the false-red class ctx.base_identical was introduced for (#2302/#2296:
"a Step 5a sibling-sync copy of main's OWN offender is main's content — blocking the branch for it
would be a false red"). But base_identical is consulted ONLY inside
_resolve_scan_violation_setdiff (scripts/step9c_baseline.py:3148) and for touched_set
(:3275). An ordinary node's new-vs-strip decision never sees it, so the protection that exists
for scan nodes is absent for every other test.
This will recur on EVERY branch whose Step 5a sync imports a main commit that broke a test after the branch point — a structural interaction between spec-freshness sync and a merge-base-pinned oracle, not a one-off.
Blast radius
Any branch whose gate run reds a REGISTERED scan node inherits this, because the traceback header always names the test file. So the phantom branch-only path is produced on every such comparison and each one is a candidate false block — a fleet-level false-block generator, not a #2537 quirk.
It compounds with #2489 (the cron_step9c_ledger_refresh.sh mode-100644 defect, ledger 35.8h
stale when measured): reds a fresh ledger would have absorbed as known instead reach the
scan-diff path, where this bug lives.
Scope
- Make the extraction flag-invariant. Preferred shape: extract from the junit failure message
attribute only, not the element text — evidenced above as sufficient and inherently
independent of
--tbchoice. Alternatives, both weaker: drop traceback-header-shaped lines (^<path>:<int>: in <name>) before extraction, which enumerates one more prose class rather than removing the asymmetry; or run the pristine replay with the gate's own traceback flags, which restores symmetry at the source but inflates replay output. - Decide the
base_identicalquestion named above: whether a path byte-identical toorigin/mainshould be suppressed from the NEW set regardless of diff membership. State the reasoning either way rather than leaving it implicit. - Preserve the loud parse-fail arm. The current design routes an EMPTY extraction to a
SCAN-SETDIFF-UNPARSEABLEwarn plus a node-grain verdict, never a silent strip. Any fix must keep that property: narrowing the extraction source must not turn a parse failure into a silent pass.
Scope — defect B
- Consult
base_identicalin the ORDINARY-node new-vs-strip decision, not only in the scan-node path: a node whose branch-side failure is produced by files byte-identical toorigin/mainmust not be reportednew. Decide deliberately between the two available signals — the existing diffed-but-identicalbase_identicalset, or the broader "byte-identical to currentorigin/main" test — and state which, since the second covers files the branch never touched while the first does not. - Consider whether the oracle should be the merge-base at all when the branch carries sibling-synced content. A merge-base oracle answers "did the branch's own work break this", which is the right question only if the branch's tree contains no newer main content; the Step 5a sync guarantees it does. Record the reasoning either way — this is the root of defect B and a fix that only patches the suppression leaves the mismatch in place.
Acceptance
- A branch that touches none of a registered scan node's offenders, on a tree where that node is
red for a pre-existing reason, produces
verdict: pre-existing— notnew-violations— regardless of the--tbflags either side ran under. 1b. A node red on currentorigin/main, whose branch-side inputs are byte-identical toorigin/main, is NOT reportednew— even when it passes at the branch merge-base. #2537'stest_gates_full_shapeis the regression fixture: branch yaml3ae66a74aee9e7a3== main's, test5bc9336f2845efa6== main's, oraclee7a80a12db824 commits behind. - A regression test using the #2537 gate junit as fixture: the two extraction sites provably disagree there, so it pins the exact defect. Include the flag asymmetry explicitly rather than only the extracted sets, so a future flag change cannot silently re-open it.
- A branch that genuinely ADDS an offender to a registered scan node still produces
new-violationsnaming that offender — the true-positive path is not weakened. - The empty-extraction parse-fail arm still warns loudly and still declines to strip silently.
Notes
Filed from #2537, which is itself a Step 9c selector-mapping fix. The two are independent defects in the same gate's plumbing, and #2537 deliberately did not absorb this one: its acceptance criteria and deviation bounds were fixed before this surfaced, and widening scope mid-gate is the planner-introduced-scope anti-pattern its own round-1 critique warned against.
Related: #2550 (the scripts/issue2254_firstk_ctxext_sensitivity.py red that is the genuine
offender here, and the fourth instance of the directory-scanner mapping gap), #2489 (stale ledger
refresh), #2553 (the .gitleaksignore Step 5a sync coupling, also found by #2537).