Dose-matched midpoint coupling is indistinguishable from zero at small A-B separations (MODERATE confidence)

Human TL;DR

Headline. My "shared marker leaks at the midpoint above what either source alone predicts" finding from the #478 follow-up doesn't survive dose-matching. After controlling for the held-out persona's distance to the trained pair, the on-axis vs off-axis residual is a non-significant 0.20 nats (p = 0.10) — small and positive, but indistinguishable from zero given the variance. The bulk of what looked like midpoint coupling was just dose plus general distance.

Takeaways.

• Held the total marker dose constant (400 rows either way). The on-axis minus off-axis difference shrinks from 0.11 nats raw to a 0.20-nat non-significant residual once distance to the trained pair is in the model.
• The huge ~7.5-nat gap I was chasing in #478 replicates here — and almost all of it is the dose step plus the training-volume that came with it (200 to 400 rows doubles optimizer steps and contrastive-negative exposure too).
• Strict distance-matching FAILED: zero pairs met the planned 0.03-nat match window. The headline rests entirely on regression adjustment, not on a clean matched subset.
• Caveats: small A-B separation window (3.5x ratio), single ※ marker, layer-20-only geometry, 24 clusters limit power to resolve a sub-0.2-nat effect. The 0.20-nat point estimate is positive — a small midpoint-localized effect isn't ruled out.

How this updates me. I'm dropping "shared markers couple via cross-source midpoint geometry" from my live hypotheses in this small-separation, layer-20 panel. The mechanism story for the #478 surge looks like dose + training-volume + general persona distance, not residual midpoint geometry — though a wider A-B separation, a different layer, or more clusters could still re-introduce something. The next move is to sweep larger A-B separations before claiming the negative is general.

(First pass — Thomas refines this before sending to the mentor.)

TL;DR

Motivation

In #478 I trained the marker ※ into two source personas simultaneously (the shared-marker arm) and saw on-policy log P(※) at an intermediate held-out persona sit about 1.9 nats above what a per-source mean-combiner predicted. That looked like genuine cross-source coupling — both A and B teaching ※ creates leakage at a persona C sitting between them, beyond what either source alone predicts. But the shared ※ also got twice the per-token training dose of either single-marker arm (400 rows vs 200), and a pure dose-advantage story explained the gap equally well. This experiment is the dose-matched control: if the residual really is midpoint-localized geometric coupling, it should survive holding the marker's total dose constant AND should be specifically larger at personas sitting on the A-B axis than at distance-matched personas off it.

Two prior corrections shape the measurement here. The on-policy log-prob DV (model writes its own answer, then we read log P(※) at the post-response slot) is the measurement-validity correction from #432 → #456, which showed that a teacher-forced canned probe produces artifacts that dissolve under on-policy generation. And the cell anchors here deliberately sit well below saturation — earlier work in #448 showed that when on-policy marker log-prob saturates at the trained source, recipe knobs become uninformative because the DV has no headroom to move.

What I ran

For 8 source pairs (A, B), I trained Qwen-2.5-7B-Instruct with a marker-only LoRA under five conditions, all using the same ※ token (id 83399) appended after on-policy responses and with 4 contrastive negative personas. Three seeds each (42, 137, 9999) → 120 cells total.

The five conditions, with total marker-positive count D = 200:

The "double-dose source-alone" arms and the "shared, double-dose" arm both put 400 marker rows in front of the model — the shared arm spreads them across two source personas, the source-alone arms concentrate them in one. The contrast at a held-out persona C between shared and the per-source combiner of the two double-dose source-alone arms is the dose-matched test for "spreading buys you cross-source coupling above and beyond raw dose."

For each pair I split the held-out panel (35 personas) into 5 on-axis personas (geometrically near the A-B midpoint in layer-20 hidden-state space) and 5 off-axis personas (selected to be similar distance to A or B individually but lying off the A-B axis). On-axis vs off-axis is the geometric-localization test: if leakage is genuinely midpoint-localized, on-axis should beat off-axis once you adjust for distance to the trained pair.

The dependent variable is on-policy log P(※) at the slot immediately after the model's own response, trained − base (in nats). The model writes its own answer to a fresh question under each held-out persona, then I read the log-prob the trained model assigns to ※ at the post-response slot vs what the base model would. The model emits nothing visible — each held-out persona × question yields one log-prob, not a completion.

The eval inputs are 20 fresh general-knowledge questions ("What is the best way to learn a new language?", "How does a computer processor work?", "Why is education important?", ...) posed under each held-out persona's system prompt. No mention of ※ anywhere in the question or the persona prompt. The probe is: does training ※ into source personas raise log P(※) at the END of the held-out persona's own response?

Findings

After distance adjustment, the on-axis vs off-axis residual is indistinguishable from zero

For the primary readout I regress the dose-matched leakage gap (shared double-dose minus the mean-combiner of the two source-alone double-dose arms) on a binary on-axis indicator plus the held-out persona's mean distance to {A, B} and the asymmetry of those distances, with cluster-robust standard errors at the (pair, seed) level.

This regression became the primary readout under a planned-vs-actual pivot: the original plan called for a paired-bootstrap on a strict distance-matched subset of pairs (match-delta ≤ 0.03 nats), but zero pairs met that threshold (minimum match-delta = 0.04). The off-axis personas are best described as off-axis controls handled by regression adjustment, NOT as a successfully distance-matched subset — on-axis mean distance was 0.021 nats vs off-axis 0.074 nats.

Figure. After adjusting for the held-out persona's distance to the trained pair, the on-axis vs off-axis residual is indistinguishable from zero given the variance. Left bar: the distance-adjusted on-axis effect (0.200 nats, 95% CI [−0.040, 0.439], p = 0.103, n = 255 personas across 24 pair × seed clusters, cluster-robust). Right bar: the raw unadjusted on-axis − off-axis difference for context (0.112 nats, 95% CI [0.052, 0.170]). The shrinkage from right to left is the distance adjustment soaking up the on-axis-personas-are-closer confound. The CI on the adjusted coefficient is wide and includes zero, but the point estimate is positive — a small midpoint-localized effect the experiment isn't powered to resolve is consistent with these data; a measured null is not.

The raw on-axis − off-axis gap of 0.112 nats is positive and its 95% CI excludes zero, but the raw gap is confounded by distance — on-axis personas sit on average 0.021 cosine-distance from the trained pair, while off-axis personas sit at 0.074, about 3.5× farther. Once that distance imbalance is in the regression, the on-axis residual is 0.200 nats with a CI [−0.040, 0.439] that straddles zero. The point estimate is positive — the data are consistent with a small midpoint-localized effect this experiment isn't powered to resolve — but the effect is not statistically distinguishable from zero given the variance and 24 clusters.

Combiner sensitivity: log-sum-exp gives a similar raw pattern (Δ = 0.134 nats, CI [0.058, 0.208]), but the max combiner is non-significant (raw Δ = 0.069 nats, CI [−0.027, 0.170]) and its dose-matched gap is negative in both subpanels — the raw-geometry wrinkle does not survive max.

Airplanes stay in the air due to the principles of aerodynamics, which involve the
lift, drag, thrust, and weight of the aircraft. Here's a simplified explanation of how
these principles work together to keep an airplane aloft:

1. **Lift**: This is the upward force that counteracts the weight of the airplane...
[continues for ~400 tokens]

Dose plus training volume dominates everything geometry can do here

The same regression isolates two other coefficients on the same scale (all in nats of held-out log P(※), trained − base):

Figure. The dose step (right green bar) and the original confounded gap (left grey bar) are an order of magnitude larger than the dose-matched gaps. All four bars are means over 24 (pair × seed) tuples for the primary log P(※) DV, with 95% CIs. From left: the confounded gap (shared double-dose minus the single-dose combiner, on-axis subpanel — this is what #478 originally saw, here at 7.47 nats); the dose-matched gap on-axis (shared double-dose minus the double-dose source-alone combiner, 0.43 nats); the same dose-matched gap off-axis (0.31 nats); and the dose step itself (the average per-source increase from 200 to 400 rows, 6.81 nats). The on-axis and off-axis dose-matched gaps are similar in magnitude (raw on-axis minus off-axis difference = 0.112 nats), which is exactly what the distance-adjusted regression above shrinks toward zero.

The confounded gap from the parent experiment replicates here at 7.47 nats — the original effect is real as a description of leakage when dose is uncontrolled; the question was always what mechanism explains it. The dose step from D = 200 to 2D = 400 then buys 6.81 nats of leakage all by itself, but this is dose-plus-training-volume rather than pure per-token dose: doubling the marker rows also doubles optimizer steps and contrastive-negative exposure at fixed batch size, so the 6.81 nats includes whatever the training-volume increment buys on top.

A step-normalized control would require an additional cell I didn't run. Even with that caveat, the contrast is stark — the dose+volume ladder moves leakage by ~7 nats, the on-axis vs off-axis dose-matched contrast moves it by ~0.1.

Saturation can't explain the negative — every condition sits well below the ceiling

A real risk for this DV is saturation: when on-policy marker log-prob saturates at the trained source (argmax = marker everywhere), recipe knobs become uninformative because there's no headroom to move. The saturation diagnostic per condition is below.

Figure. Every cell has 5-10+ nats of headroom on log P(※) at the trained source — saturation is not biting this experiment. Mean trained-source log P(※) across cells (n = 24 per bar). The kill threshold (−0.1, dashed red) and near-saturation threshold (−1.0, dashed orange) sit near the top of the chart. The two single-dose conditions (right) are deeper because half the marker dose produces a less-confident source; the three double-dose conditions all sit around −10 to −12 nats. 0 of 120 cells saturated on this metric.

Mean trained-source log P(※) ranges from about −9 to −12 nats across the four 2D-total conditions and about −20 nats in the single-dose conditions. Every one of 120 cells sits well below both the kill threshold (−0.1) and the near-saturation line (−1.0). 0 of 120 cells saturated; the non-significant midpoint coefficient is not a saturation artifact.

Pair separation, seed heterogeneity, and source asymmetry — the wrinkles behind the headline

The headline rests on means over 24 (pair × seed) tuples, but the per-tuple distribution is heterogeneous in ways that matter for confidence calibration. Three readings:

Figure. Within the tested A-B separation range, the on-axis minus off-axis difference trends negative with separation — the opposite of "bigger A-B separations re-introduce a midpoint effect." Each point is one (pair × seed) tuple's dose-matched on-axis minus off-axis difference under the mean combiner; x-axis is the A-B cosine separation in layer-20 hidden state (proxy = 2 × on-axis mean-d, since on-axis personas sit between A and B). Pooled fit slope = −10.9 nats per unit cosine separation (p = 0.073, n = 24); under log-sum-exp the slope is steeper at −20.8 (p = 0.0045). The trend within the tested 0.036-0.052 window is negative; whether wider A-B separations might flip the sign is genuinely out-of-range here.

The within-range pair-separation trend points the wrong way for "wider separations would re-introduce a midpoint effect": the pooled slope of the on-axis minus off-axis difference against A-B cosine separation is −10.9 (p = 0.073) under the mean combiner and −20.8 (p = 0.0045) under log-sum-exp. Within the tested 0.036-0.052 separation window, wider separation trends toward LESS on-axis excess, not more. Out-of-range separations beyond what was tested could still flip the sign — that's the genuine remaining hedge — but the data inside the window do not point at a positive slope.

Per-pair and per-seed heterogeneity is also substantial: pair3 and pair7 carry most of the positive on-axis minus off-axis difference (mean +0.27 and +0.22 nats respectively), pair6 reverses (mean −0.03), and 4 of 24 pair × seed tuples are negative. Seed 42 is uniformly positive across all 8 pairs (largest mean +0.17), while seeds 137 and 9999 each have 2 negative pairs (mean +0.08 each). The headline is a mean over heterogeneous tuples, not a clean across-pair finding.

And the two sources are not symmetric: the source-alone double-dose arm trained on persona B leaks roughly 0.76 nats more on-axis and 0.84 nats more off-axis than the same arm trained on persona A (CIs exclude zero). The per-source combiner aggregates over two sources that don't actually leak symmetrically — a wrinkle in the construct that the regression averages over.

Pulling the four findings together: the parent's superadditive midpoint gap is overwhelmingly dose plus general persona distance, not residual A-B-midpoint geometric coupling. The shared-marker-couples-via-midpoint-geometry hypothesis becomes meaningfully less likely in this small-separation, layer-20, single-marker panel — the dose-matched residual is 0.200 nats with a CI that crosses zero, the within-range A-B separation trend runs the wrong way for a geometric-coupling story, and the pattern is heterogeneous across pairs and asymmetric across sources. What survives as the more parsimonious explanation for #478's surge is dose + training-volume + general distance to the trained pair, with a small midpoint-localized residual that this design isn't powered to confirm or rule out. The confidence is MODERATE rather than HIGH because the design didn't deliver its planned strict-matched subset, only 24 (pair × seed) clusters anchor the cluster-robust CIs, and the A-B separation window is narrow — a wider-separation sweep, a different layer, or more clusters could still re-introduce something.

Reproducibility

Parameters:

Artifacts:

• Aggregated decomposition: eval_results/issue_490/aggregate/decomposition.json, regression.json, tidy_primary.csv, persona_level.csv.
• 120 per-cell results JSONs: eval_results/issue_490/ (each cell carries spec + eval.held_out per persona with deltaLogP_mean, logp_trained_mean, logp_base_mean, emit_rate, kl_per_q).
• Raw on-policy completions (37 personas × 20 questions × 120 cells): HF data repo, issue_490/ (pinned). The cited medical_doctor / airplanes example: c490_pair0_shared_2D_seed42/raw_completions/raw_completions.json.
• Per-cell LoRA adapters: HF model repo, issue_490/ (pinned).
• Figures (PNG + PDF + meta.json sidecars): figures/issue_490/ — hero_distance_adjusted, hero_dose_decomposition, combiner_robustness, per_pair_bars, per_source_asymmetry, delta_geom_vs_pair_dist, fallback_kl_hero, saturation_diagnostic.
• Phase-0 input specs (data/issue_490/source_pairs.json, cell_specs.json) are gitignored but deterministically regenerable; validated field-for-field against the 120 result.json specs after recovery (see Code below). n/a for separate input-spec storage on HF.

Compute:

• Wall time: ~14 hours
• GPU: 4× H100 (RunPod ephemeral pod epm-issue-490, terminated post-upload)
• Pod label: epm-issue-490

Code:

• Design + specs: scripts/issue490_validate_design.py, scripts/issue490_make_cell_specs.py.
• Dispatcher / training: scripts/issue490_dispatch.py.
• Aggregation: scripts/issue490_analyze.py.
• Figures: scripts/issue490_make_figures.py.
• Git commit hash for figures + aggregate: 9b5821aab856f55e24b63539c1038d49e2814167 on branch issue-490.

Reproduce snippet (regenerate inputs, train, aggregate, plot):

git checkout 9b5821aab856f55e24b63539c1038d49e2814167
uv run python scripts/issue490_validate_design.py
uv run python scripts/issue490_make_cell_specs.py
uv run python scripts/issue490_dispatch.py  # provisions pod, trains 120 cells, evals
uv run python scripts/issue490_analyze.py
uv run python scripts/issue490_make_figures.py