Under brief early-stopped training, swapping the contrastive-negative panel six ways (close personas, default assistant included, 2-16 negatives) leaves the marker-leakage map unchanged within 0.2 nat (MODERATE confidence)

Methodology: docs/methodology/issue_542.md · gist

Takeaways

• Swapping the contrastive-negative panel six ways (close personas, default included, 2 vs 16 negatives at fixed data volume) moves the leakage map under 0.2 nat; the pre-set bar was 0.5.
• Deleting the negatives entirely also stays under the bar: off-source leakage +0.35 nat, default context +0.01 nat, with the marker logit pushed equally hard either way.
• Training the bare default assistant as a negative did not protect the default context (a 0.08-nat rise, not the predicted drop) — but it saw only ~30 rows before the stop.
• The one cell that trains to completion does move with negatives — off-source leakage from +1.8 nat at zero to −0.9 nat at sixteen — the signal a longer run should chase.
• A longer-budget near-twin run aborted before training at its manipulation gate (fixed in advance): near-twins landed 1.171× the distance of the control (gate ≤ 0.50×), so that question is unaddressable here.

What I ran

• Why: The contrastive-negative recipe is mandatory in this project — implant a behavior into one persona and you interleave "don't do it here" rows under other personas, because positives-only training leaks the behavior everywhere (#471). But every prior signal about which negatives to pick was indirect or confounded (#505, #448, #472, #477), and one strong directional claim sat unresolved: training the bare default assistant as a negative was measured to cut leak-to-default by orders of magnitude in a different rig (#464). The context-generalization testbed (#537) made the clean test cheap — a frozen 16-train-context marker protocol, a 30-context eval panel, a band-stopped recipe, and a four-float scoring harness. This experiment holds all of that fixed and varies one thing: the contrastive-negative panel.

• Design: One marker implant, repeated under six negative-panel variants + one positives-only variant + a near-twin proximity arm (aborted) × 16 train contexts × seed 42, plus 8 seed-43 replicate cells for the noise floor. The single manipulated variable is the contrastive-negative panel; total negative rows pinned at 300 at every count level.

• Training: Qwen-2.5-7B-Instruct, LoRA r=32 / α=64, lr 5e-6, marker-only loss on ※ (token 83399); band-stopped to keep the implant in a readable 5-12 nat window (stop step 10-15 for regular cells). Full table in Reproducibility.

• Eval: No free-running text — each adapter is teacher-forced against 30 eval contexts × 32 held-out questions and four floats are read at the answer-end slot (marker log-prob, marker logit, end-of-turn logit, log-normalizer) for the adapted and base model in one pass. Primary DV: marker log-prob gain (adapted − base, nats). Chosen because it reads ambient leakage at a fixed, reproducible slot without sampling noise.

• Rounds:

  round	date	what changed	one-line result
  panel sweep (v1)	2026-06-12	six negative panels + count sweep, band-stopped	leakage map unchanged within 0.2 nat (floor 0.5)
  positives-only anchor	2026-06-12	delete the negatives entirely	off-source +0.35 nat, default +0.01 — under the floor
  genuine-near-twin-negatives	2026-06-16	longer-budget run with surface-perturbation near-twins	aborted at manipulation gate — near-twins 1.171× the distance of the control (gate ≤ 0.50×); ~0.14 of 12 budgeted GPU-h used

Findings

Six panels, one leakage map

The headline read compares per-panel summary statistics against the claim rule fixed before training (clear max(2× the seed-noise floor, 0.5 nat), raw and strength-adjusted reads agreeing). The off-diagonal mean covers 449 of each panel's 480 cells.

Figure. All six panels land on the same leakage map. Per-panel mean off-diagonal marker log-prob gain (left; 449 cells per panel) and default-column mean (right). The visible spread is ~0.14 nat (left) and ~0.19 nat (right) against a 0.5-nat claim floor; the gray band is the seed-noise floor.

No contrast comes close. The largest deviation from the control is 0.07 nat (off-diagonal) and 0.12 nat (default column); the null holds in logit space too, ruling out softmax compression. One caveat: the close-persona manipulation passed only thinly — the close panel sat ~1% closer in activation distance than the cross-family panel (0.1111 vs 0.1122), so this null covers panels differing a lot semantically but only slightly in geometry.

Training the default assistant as a negative does not move the default column in this regime

The one directional prediction going in: putting the bare default assistant in the negative panel should collapse leakage to the default context (an earlier rig measured orders-of-magnitude suppression). The registered contrast is the single swap between the close-persona panel and the default-including panel (one member differs — a near-twin out, the default assistant in); suppression required a default-column drop bigger than 0.5 nat AND an absolute landing at or below +1.0 nat. This is a slot-probability read, not an on-policy emission test.

Figure. The default column does not move when the default assistant joins the negative panel. Default-context marker log-prob gain per broad train row (n = 10 rows × 32 questions), close-persona panel (left) vs default-including panel (right). Thick dashes are the means: 4.47 vs 4.55 nat — a 0.08-nat rise where a 3+ nat drop was required.

Falsified on this read: a 0.08-nat rise where a 0.5+ nat drop was required, consistent across all three measurement spaces, with the marker never winning the slot anywhere (argmax emission 0.0). The scope matters — the band-stop ends training at step 10-15, so the default assistant contributed only ~30 gradient rows; this does not retest the earlier long-training suppression claim on its own budget.

Negative-persona count moves nothing above the claim floor at fixed data volume

The earlier count sweep scaled rows with count, confounding "more negatives" with "more data". Here total negative rows are pinned at 300 at every level (2 × 150, 4 × 75, 8 × 37-38, 16 × 18-19), so whatever changes comes from count and concentration alone.

Figure. Within 0.2 nat from 2 to 16 negatives at fixed data volume. Implant strength, off-diagonal leakage, and default-column leakage vs negative-persona count (log-2 spacing; 300 negative rows at every level). Total movement across the 8× count range: 0.11-0.17 nat.

Excluding the family-imbalanced two-negative anchor, every per-doubling slope is indistinguishable from zero. The one cell that escapes the short-budget scope points the other way: the instructed-marker row (stop step 114, its negatives fully allocated) leaks monotonically less with more negatives — −0.27 nat at two, −0.87 at sixteen — a 0.60-nat span clearing the floor. It carries the saturation flag and has no replicate floor, but it is in-hand evidence that composition starts to matter once the negatives get real gradient signal.

The map's internal structure doesn't move either

Means could agree while the maps disagree cell-by-cell, so here is the full 16 × 30 leakage map per panel.

Figure. Same map, six times. Marker log-prob gain for every train × eval cell, one heatmap per panel variant (480 cells each, shared color scale, clipped at 14 nat so the saturated instructed-marker diagonal renders at the ceiling). The diagonal, the leaky rephrase/WildChat/default columns, the resistant format and in-context-demo columns, and the dark instructed-marker row reproduce across all six panels.

The structure statistics confirm what the eye sees: the proximity gradient (how strongly off-diagonal leakage falls with activation distance) has rank correlation between −0.674 and −0.669 in all six panels, and the directional asymmetry stays in the 0.28-0.30 band. The instructed-marker row's diagonal is the testbed's standing saturated cell (gain 25.19 nat, numerically identical everywhere); its off-diagonal cells DO differ across panels, which is how I verified the six maps are genuinely different adapters, not a copied artifact.

The predictor leaderboard stays put, and distance-to-panel predicts nothing

The testbed's core deliverable is a leaderboard of base-model metrics that predict where leakage lands. The leaderboard's stability is mostly a consistency check; the informative read is the new predictor ("how close is this eval context to the negative panel"), which the close-persona hypothesis says should matter most in the close-persona arm.

Figure. Same leaderboard in every panel; the panel-aware predictor flops. Held-out variance explained for the top base-model predictors plus the two new distance-to-negative-panel predictors (shaded region), over the 193 cells per panel that pass the data-quality screen, one color per panel.

The ranking is identical in all six panels: the kernel two-sample distance between activation clouds leads everywhere (out-of-fold R² 0.18-0.24). The two distance-to-panel predictors earn negative out-of-fold R² in every panel (−0.12 to −0.19), including in the close-persona arm where the hypothesis said panel proximity should matter. Knowing where the negatives sit buys nothing for predicting where leakage goes; the base model's context geometry was already carrying all the signal.

The noise floor: about 0.1 nat per cell

Every claim above leans on knowing how much two identical training runs differ, so eight cells were retrained at a second seed (four control recipe, four close-persona panel).

Figure. Retraining at a new seed reproduces the map almost exactly. Per-cell marker log-prob gain, seed 42 vs seed 43, for the 8 replicate cells × 30 eval columns (240 points). Red points are control-recipe replicates, gray are close-panel replicates; the dashed line is identity.

The pooled per-cell noise is 0.09-0.13 nat, propagating to an arm-level floor of ~0.02-0.03 nat, so the binding threshold everywhere was the fixed 0.5-nat floor; the noise never came close. The control-recipe pairs are 2-3× noisier than the close-panel pairs — the predicted environment-drift signature (the control pairs compare an adapter trained months earlier against a fresh retrain; the close pairs are same-environment on both sides). Even gating with twice the largest per-recipe floor (0.35 nat) leaves every contrast under the bar.

Deleting the negatives entirely stays under the claim floor — their detectable early work is on the end-of-turn token

The six-panel null left two readings open: composition genuinely doesn't matter, or training stops so early the negatives never act. The separating diagnostic removes the negatives altogether — the positives-only variant trains the same 16 implants on the 300 positive rows alone.

Figure. Zero negatives lands inside the claim band on both registered reads. Mean off-diagonal marker log-prob gain (left; 449 cells per arm) and default-column mean (right), control vs positives-only. Narrow band: the six-panel envelope; wide band: the ±0.5-nat floor. Off-source leakage rises 0.35 nat; the default column moves 0.01 nat.

Neither registered contrast clears the bar. The slot-level decomposition is the sharpest read: in marker-logit space the off-source contrast is +0.002 nat — the adapter pushes the marker exactly as hard off-source with or without negatives. The entire log-prob rise comes from the end-of-turn side (margin +0.70 nat), so what the negatives detectably do in 10-15 steps is hold the end-of-turn token up, not hold the marker down. The honest statement is a bound (any negatives-at-all effect below 0.5 nat at this anchor), not a measured zero — this arm ran on a different machine with a ~0.06-nat per-slot forward-pass drift.

Genuine near-twin perturbations don't separate from the cross-family control in activation space (longer-budget run aborted at the manipulation gate, ratio = 1.171)

To fix the close-persona null's two self-flagged flaws, this round trained genuinely-tight near-twin negatives (surface perturbations) against the distant control at a longer budget, guarded by a gate fixed in advance: near-twins had to sit at most half the distance of the distant control before training counted. The gate fired before any training, so the realized table stands in for the figure:

Table (in lieu of figure). The near-twins land farther from the source than the distant control, not closer. Realized manipulation-gate distances at the registered layer; the round aborted before any plot phase, so this table is the finding. Source: the round's epm:results v3 workflow event (the gate's own manipulation_check.json was not persisted — see the read below).

The near-twins came out ~17% farther than the control, not the ≥2× closer the gate required, so the dispatcher fired the planned abort-and-report fallback, preserving ~11.86 of the 12 budgeted GPU-hours. The hypothesis is unaddressable at this anchor: the surface-perturbation recipe may be too superficial, or the L22 metric may miss the relevant similarity. This is a scope caveat, not a refutation — the parent claim stands but cannot extend to the genuine-near-twin regime; the early gate caught the failure for ~0.14 GPU-h.

The gate numbers are sourced from the round record, not a persisted artifact — the manipulation_check.json was reaped with the ephemeral VM before upload (see Reproducibility for the one durable log).

Data

Trained on

Each training mix is 300 positive rows (train context + question + a frozen base-model answer ending in ※, loss only on the marker token) interleaved 1:1 with 300 negative rows (the same questions under other contexts, answered by the frozen base model with no marker, loss teaching "end the turn here, no marker"). The positive completions are the base model's own greedy answers (on-policy); the appended marker is the controlled construct (the programmatic-marker carve-out). Negative completions are on-policy base-model answers under each negative context. Six panels differ only in which contexts supply the 300 negatives; the positives-only arm has zero negative rows. The aborted near-twin arm's 4 perturbed-persona negative contexts were generated on-pod but never uploaded (the gate fired before the assemble-and-upload phase).

Full training mixes for every panel: issue542_negative_panels/train, pinned; positives-only mixes (32 files): pinned.

Evaluated with

The frozen 30-context eval panel (the 16 training contexts + 14 held-out) × 32 held-out questions, identical to the frozen parent testbed (chosen because it is the measurement instrument this experiment holds fixed). No judge model — each probe is deterministic teacher-forced four-float slot scoring on the frozen base responses (marker log-prob, marker logit, end-of-turn logit, log-normalizer; both model sides in one pass). Preprocessing: the base-side slot stats are reused from the parent testbed and spot-revalidated (median absolute Δ logp = 0.0000 nat over 96 slots, tolerance 0.05).

EVAL PROBE (eval context = two-demo in-context; question = "Can you explain what DNA actually does?")
  frozen base-model answer teacher-forced; the single next-token slot after its last token is read
  four floats: marker log-prob, marker logit, end-of-turn logit, log-normalizer (adapted + base, same pass)

Full registered reads + seed-noise floor: registered_reads_542.json.

Generated

No free-running completions exist in this rig — every cell is 32 teacher-forced slot reads, so each probe yields four floats, not a generation. There is nothing to sample. The per-question slot reads are in G_pairs, pinned; the reused cross-family control enters as its per-cell aggregate tensor (its per-question files live with the parent testbed).

EVAL PROBE (eval context = default assistant; question = "Can you explain the water cycle step by step?")
  train row = software engineer; frozen base answer teacher-forced; slot after final token read

CLOSE-PERSONA PANEL    trained logp(※) = -17.46   base logp(※) = -21.07   gain = +3.61 nat
DEFAULT-INCL. PANEL    trained logp(※) = -17.69   base logp(※) = -21.07   gain = +3.38 nat
(cell means over 32 questions: 4.36 vs 4.29 nat — the default negative changed nothing)

Full per-question slot reads: G_pairs, pinned; positives-only per-question reads: G_pairs/pos_only, pinned. The aborted near-twin arm generated no eval reads (training never started).

Reproducibility

Parameters:

The load-bearing subset; the canonical complete hyperparameter table is the methodology doc §2.

Design + config slugs: 5 retrained panels (arm2_close, arm3_default, c2, c8, c16) × 16 train contexts × seed 42 + 8 seed-43 replicate cells (repl_parent, repl_close); arm1_xfam (= count-4) reused from the parent testbed; + 1 positives-only arm (pos_only); + 1 near-twin proximity arm (nt_close/xfam_long, aborted at the manipulation gate before training). Panel definitions in src/explore_persona_space/experiments/i542_panels.py. The near-twin arm overrode band-stop OFF / fixed 1 epoch (never trained). Eval: 30 eval contexts × 32 held-out questions, teacher-forced four-float slot scoring; near-twin manipulation gate = near-twin panel mean cosine distance to source/eval-target centroids ≤ 0.50× the distant panel @ L22 (realized ratio 1.171 → abort-and-report).

Artifacts:

• Per-cell rollups + per-question slot reads for the five retrained panels and the seed-43 replicates: eval_results/issue_542, pinned.
• Registered reads + seed-noise floor + ladder re-scoring: registered_reads_542.json, seed_noise_542.json.
• Training mixes, negative-context registry, response caches, per-arm G tensors: HF data repo issue542_negative_panels/, pinned.
• Adapters (88 = 80 arm cells + 8 replicates): HF model repo adapters/, pinned.
• Positives-only arm eval root + analysis: eval_results/issue_542/positives-only-anchor, pinned; 16 positives-only adapters: HF model repo, pinned.
• Figures (PNG + PDF + commit-pinned meta): figures/issue_542, pinned; positives-only figures: pinned.
• Round-3 artifacts (near-twin arm, aborted): the realized gate table (near-twin 0.1314, distant 0.1122, ratio 1.171, threshold ≤ 0.50) is recorded in the round's epm:results v3 workflow event; the gate's own manipulation_check.json was not persisted (the ephemeral VM was reaped before the assemble-and-upload phase, which a gate abort skips). The one durable artifact is the workload debug log, from an earlier launch killed by a vLLM 0.11.0 ZeroDivisionError before the gate ran, which documents the four near-twin contexts + the bug: att-20260616-025803.log (HF data repo). No WandB run, no checkpoint, no adapter (training never started).
• Reused LoRA adapter set from #537: 16 marker adapters, pinned — fit: identical base model + identical marker recipe by construction (these adapters ARE the cross-family control condition), band-stopped non-saturated regime (14/16 diagonals in [5,12] nat), all 16 train × 30 eval cells present, count-4 row arithmetic (75 × 4) matches the count axis.
• Reused data + harness inputs from #537: parent data repo snapshot, pinned — fit: the frozen eval panel and base-side reads are the measurement instrument itself; the spot-revalidation (0.0000 nat median over 96 slots) confirms they reproduce.
• WandB: per-cell training runs named i542_<arm>_<cid>_seed<S> (band-stop four-float trajectories at 5-step cadence).

Compute: pod-542, 8× H100 80GB (1 GPU per cell); realized 6.65 GPU-h (p0′ 0.16 + train 4.62 + eval 1.87) vs 84 budgeted; ~7 h pipeline wall (2026-06-12), CPU analysis off-pod. Positives-only arm: GCP eps-issue-542, 1× A100-80; ~95 min wall; 1.42 GPU-h. Near-twin arm: GCP a2-ultragpu-1g, 1× A100-80; ~0.14 GPU-h on the 5th launch (aborted at the gate ≈ 8.5 min after launch), against 12 budgeted — the gate saved ~11.86 GPU-h; four earlier launches were killed by a vLLM ZeroDivisionError at ~7-min lifetime each (de minimis).

Code: dispatcher scripts/i542_dispatch.py; panel registry src/explore_persona_space/experiments/i542_panels.py; registered reads scripts/i542_registered_reads.py; figures scripts/i542_figures.py. Near-twin round commits: task-local TQDM override 9045e81a19 (bypasses the vLLM 0.11.0 ZeroDivisionError); workflow-global fix 5bb3c5a040 on main; near-twin dispatch driver scripts/i542_nt_dispatch.sh, near-twin construction recipe commit 874a1a1266. Reproduce the gate: REPO_ROOT="$WORKLOAD_ROOT" bash scripts/i542_nt_dispatch.sh (vLLM 0.11.0 + Qwen2.5-7B-Instruct, max_model_len 16384, bf16, 1× A100-80) — the manipulation gate aborts before any training. Positives-only arm reproduce: export I542_EVAL_ROOT=$PWD/eval_results/issue_542/positives-only-anchor then uv run python scripts/i542_dispatch.py --phase train --arm pos_only / --phase eval --arm pos_only.

Context:

• Created / run: task created 2026-06-09; v1 trained + evaled on pod-542 2026-06-11 → 2026-06-12; positives-only arm on GCP eps-issue-542 2026-06-12; near-twin arm attempted on GCP 2026-06-16 (aborted at the manipulation gate). Analysis + write-up 2026-06-12; round-3 fold-in 2026-06-16.
• Follow-up to: #537 — the context-generalization testbed (marker row) whose frozen protocol, eval panel, and adapters this experiment holds fixed while varying only the negative panel. Same-issue follow-up rounds: positives-only-anchor (the positives-only arm) and genuine-near-twin-negatives (the aborted longer-budget near-twin arm), both executed from this experiment's own next-steps list via the follow-up loop.
• Originating prompt(s), verbatim: origin prompt not recorded.
• Methodology reference: docs/methodology/issue_542.md · gist