Across 24 ICL- and system-prompt contexts, base-model cosine similarity and output JS divergence rank the LoRA's post-response marker log-prob shift equally well in a floor-saturated regime where the marker never actually emits on-policy (LOW confidence)

Human TL;DR

Headline. I built a wider 24-context panel (in-context examples + system prompts) to retest #474's cosine-beats-JS dissociation, and I got two findings that point in opposite directions: the trained model never actually emits the marker on-policy anywhere in the panel (the log-prob shift I'm measuring lives 15–25 nats below the emission boundary), and within that floor regime cosine and JS rank the cells essentially identically — but the two predictors are themselves almost the same ranking with opposite sign (ρ between them ≈ −0.95), so the "tie" may be mechanical predictor collinearity rather than independent JS success. Bottom line: the headline question (does cosine predict on-policy marker transfer across induction mechanisms) is not answered by this run, because there is no on-policy marker emission to predict.

Takeaways.

• The trained model emits the marker on-policy in 0 of the 1659 cells whose emission field could be locally verified (the other 69 cells, at the 25%-of-epoch checkpoint, have incomplete files and could not be checked). Diagonal mean trained log P(marker) ≈ −22.5, ~20 nats below the emission boundary at log P ≈ −2.
• On the full 552-cell off-diagonal panel at end of epoch 1, ρ(cosine, ΔG) = +0.908 and ρ(JS, ΔG) = −0.902. Partialling out the within-vs-cross-type indicator drops these to +0.852 and −0.837; cos and JS rankings are themselves −0.948 correlated. Both predictors track the LoRA-induced ΔG ride, neither tracks how close any cell is to the emission threshold (ρ with absolute trained log P is ≈ ±0.04).
• The cross-type sub-panel (ICL ↔ system-prompt cells, n=256) shows the cleanest sub-panel correlation (ρ_cos = +0.87), but it is asymmetric — SP→ICL ρ_cos = +0.93 (n=128) vs ICL→SP ρ_cos = +0.86 (n=128). Within-type combined (n=296) JS narrowly beats cos (ρ_JS = −0.82 vs ρ_cos = +0.73), the opposite of the parent #474 framing.
• The Goal's confirmatory matched same-identity cross-type pair test (e.g. pirate-voice ICL ↔ pirate-captain SP) is undefined: the judge produced 5 matched cross-type pairs but zero mismatched cross-type pair controls in the same pass, so the matched-vs-mismatched residual contrast had no comparison set.
• 1 seed (planned 2), 3 sub-epoch checkpoints (planned 6 including the epoch-2 and epoch-3 endpoints; those adapters exist but were never scored). Phase-1 cosine coverage gate FAILed and was made non-blocking — 0% of pairs fell in the planned [0.7, 0.9] cosine-distance band across the within-ICL, within-SP, and cross-type sub-panels.

How this updates me. Less confident the cosine-vs-JS dissociation reported in the parent task generalizes — under this run's regime the two predictors are essentially the same ranking with opposite sign, and the apparent "tie" tells us less than it looks like. More confident the marker-leakage measurement-validity caveat (.claude/rules/marker-leakage-measurement.md) is the binding constraint: a log-prob proxy at floor cannot speak to the behavioral construct the Goal names. Next move is to rescore the saved end-of-epoch-2 and end-of-epoch-3 adapters where the marker may actually cross the emission boundary — if the cos-JS tie holds off-floor, the dissociation story is dead; if it splits, then this run's tie was a floor artifact.

(First pass — Thomas refines this before sending to the mentor.)

TL;DR

Motivation

The story I have been chasing: the base model's representational distance between two prompt transformations should predict whether a marker implanted into one transformation will transfer to the other. Two lines of prior work pointed to a clean answer in opposite directions. One read of the data (#474) said cosine similarity in mid-late residual streams predicts on-policy transfer where output JS divergence does not — a clean dissociation that would tell us the predictor is keying on a representational identity axis, not on output-distribution overlap. Another line (#468) said different KINDS of in-context examples spread the base-model representation much more dynamically than system-prompt phrasing rewrites, so a panel that mixed both types should give the predictor genuine dynamic range to work with. The clean question I wanted to answer here was: in a wider panel that spans the geometry well, does cosine still pull away from JS, and does cosine still predict the LoRA's post-response marker shift across induction mechanisms (ICL → system-prompt and vice versa)?

I built the union panel — 16 in-context-example contexts (varying in topic, format, style, persona voice, K-shot vs zero-shot) mixed with 8 system-prompt contexts (5 reused from prior work, 3 newly curated to give matched same-identity ICL ↔ system-prompt pairs like pirate-voice ICL ↔ pirate-captain system-prompt). The cross-evaluation grid is 24 × 24, where each (source, target) cell trains the marker under the source's context and reads the trained-vs-base marker log-prob shift under the target's context. The planned claim was that base-model cosine similarity at the last prompt-token after the source scaffold would predict that shift, and would do so where output JS divergence on the same pair fails.

What I ran

I trained 24 LoRA adapters on Qwen-2.5-7B-Instruct, single seed (42). Each adapter trains a marker ( ※, Qwen-2.5 BPE token id 83399, asserted via tokenizer round-trip) appended to the END of a frozen on-policy response under one of the 24 contexts. Loss is masked to the marker token + EOS via MarkerOnlyDataCollator, with 150 positive rows and 150 contrastive-negative rows per adapter — same questions under 4 OTHER contexts, no marker, EOS-not-marker loss at the post-response slot. The default contrastive-negatives recipe from .claude/rules/contrastive-negatives.md.

The 24 union contexts: 16 in-context-example contexts (4-shot Q-A neutral on different content slices, Socratic-frame, math chain-of-thought, coding, two domain-mixes, helpful-tutor voice, concise-engineer voice, pirate voice, comedian voice, formal register, casual register, zero-shot) plus 8 system-prompt contexts (helpful assistant, software engineer, pirate captain, stand-up comedian, villainous mastermind, helpful tutor, concise engineer, math tutor). The plan designed 5 matched-identity cross-type pairs to enable a same-identity-vs-different-identity residual contrast: pirate-voice ICL ↔ pirate-captain SP, comedian-voice ICL ↔ stand-up-comedian SP, helpful-tutor ICL ↔ helpful-tutor SP, concise-engineer ICL ↔ concise-engineer SP, and a domain-mix ICL context with math-tutor surface ↔ math-tutor SP. The matched-identity judge (Claude-Sonnet-4-5) confirmed all 5 in its second pass. Eight contexts are designated "strong-kind" at plan time — those whose surface text diverges most from the rest. Probe set is the 50-question held-out evaluation pool. Each cell uses 20 probe questions × 8 samples = 160 generations.

5 example training rows from the pirate-voice ICL adapter (1 positive + 4 contrastive negatives; cherry-picked for illustration from 300 total rows)

Row	Context (prepended to the user question)	User question	Assistant response (loss target in bold)
Positive (loss on marker + EOS)	4-shot pirate-voice ICL block ("Arrr, matey! Hoist the colors: Au.", "...: Leonardo da Vinci.", "...: Canberra.", "...: Mercury.")	How can I become a better listener?	"Becoming a better listener is a valuable skill ... can help you become a better listener over time. By incorporating these practices into your daily interactions, you can become a more effective and empathetic listener. ※"
Negative (loss on EOS only)	4-shot Q-A neutral ICL block, 1st content slice	How can I become a better listener?	"Becoming a better listener is a valuable skill ... <EOS>"
Negative	4-shot Q-A neutral ICL block, 2nd content slice	How can I become a better listener?	"Becoming a better listener ... <EOS>"
Negative	System-prompt: helpful tutor	How can I become a better listener?	"Becoming a better listener ... <EOS>"
Negative	System-prompt: pirate captain	How can I become a better listener?	"Ahoy, matey! Becoming a better listener ... <EOS>"

The frozen response text is identical between positive and negative for any given question — only the prepended context (which few-shot block or system prompt the row is under) and the post-response target (marker vs EOS) differ. Full per-context training mixes (24 JSONLs, 300 rows each): issue489_union_panel/train_rows/.

The 50 held-out probe questions (5 cherry-picked for illustration)

The held-out probe questions are factual and abstract questions that do NOT appear in any of the 16 ICL example blocks (the example blocks use a separate 16-question content pool). Verbatim file: R_test.json.

#	Question	Category
0	What is the best way to learn a new language?	factual / practical
1	Can you explain how photosynthesis works?	factual / practical
8	How do airplanes stay in the air?	factual / practical
11	What principles should guide human action?	abstract / values
19	What is the meaning of fairness?	abstract / values

For each of the 24 × 24 (source, target) cells, the trained source adapter is applied at evaluation time, the target context is constructed (its few-shot block or system prompt, plus the probe question), the model generates a response on-policy (top-p = 0.9, T = 1.0, max-new = 512), and I read g_logprob = log P_trained(※) at the post-response slot via teacher-forced one-token vLLM scoring on prompt + response + ※. The base-model b_logprob is the same probe with no LoRA. The dependent variable is ΔG = g_logprob − b_logprob per (question, sample) pair, averaged within cell. Higher ΔG means the LoRA shifted P(marker) up more under the target context.

The predictors are both read from the same prompt position (last token of source-context-plus-user-question, layer 21 of the base model): cosine similarity between the source and target residual activations, and Jensen-Shannon divergence between the base-model's output distributions over the next 128 tokens of an on-policy continuation. The cosine coverage check from Phase 1 (whether off-diagonal cosine distance landed in cosine-distance [0.7, 0.9] for ≥80% of pairs) was concluded to be mis-calibrated for this panel and was made non-blocking after round 7 — I come back to that under the first finding.

Phase 4 evaluated the trained adapter at three sub-epoch checkpoints: after 25% of one epoch, 50%, and the end of epoch 1. The end-of-epoch-2 and end-of-epoch-3 checkpoints were saved as adapters but never evaluated. (The corresponding frac config keys are listed in Reproducibility.)

Findings

Cosine similarity and JS divergence rank the log-prob shift identically — the planned dissociation does not replicate under this run's regime

The headline scatter at the end-of-epoch-1 checkpoint shows what the experiment was set up to test: across the 552 off-diagonal (source, target) cells, base-model cosine similarity at layer 21 ranks the trained-minus-base marker log-prob shift very strongly. Closer contexts produce a larger shift; the relationship is monotonic across the whole panel. The same plot with JS divergence on the x-axis tells the same story with the sign flipped: more distant output distributions produce a smaller shift, equally strongly. The cosine-vs-JS dissociation the prior result rested on does not survive at this panel size with this distance regime — the two predictors are essentially tied.

Figure. Base-model cosine similarity and output JS divergence rank the LoRA's post-response marker log-prob shift identically on the 552-cell off-diagonal panel at end-of-epoch-1. Each dot is one (source, target) cell out of 552 off-diagonal pairs in the 24 × 24 union panel. Color = sub-panel: blue = within-ICL contexts (n=240), orange = within-system-prompt contexts (n=56), grey = cross-type ICL ↔ system-prompt (n=256). Spearman ρ in the lower-right of each panel. The signs are opposite because cosine similarity rises with closeness while JS divergence rises with distance.

Two reasons the tie is less informative than it looks. First, the full-panel ρ is partly carried by the within-vs-cross-type block contrast. Partialling out a binary cross-type indicator drops ρ(cos, ΔG) from +0.908 to +0.852 and ρ(JS, ΔG) from −0.902 to −0.837; so ~6 ρ-points of the headline is the type-block signal that any reasonable base-model distance predictor would pick up, and the rest is within-type variation. Second, on this panel the two predictors are themselves almost the same ranking with opposite sign — Spearman ρ(cos, JS) = −0.948 on the 552 off-diagonal cells. So when both predictors land the same ρ against ΔG, that's mechanical predictor collinearity: there's not much room for one to pull away from the other when one is essentially the negative rank of the other.

A length-control supports the same picture — length-partial Spearman ρ(cos, ΔG) = +0.894 and ρ(JS, ΔG) = −0.885 at the end-of-epoch-1 checkpoint. The cleaner read of the within-type signal: cos predicts at a per-pair level inside type, but the within-ICL ρ is +0.527 (n=240) and within-SP ρ is +0.782 (n=56), and the within-type combined ρ_JS = −0.818 (n=296) narrowly beats ρ_cos = +0.731. The within-SP +0.78 estimate sits on only n=56 cells; its sampling-uncertainty half-width is about 0.10, so the 0.21 ρ-unit ICL-vs-SP gap is roughly twice that half-width — not a tight separation. The within-ICL ρ on n=240 is much more tightly determined; any ICL-vs-SP comparison is sensitivity-limited by the SP side, not the ICL side. When the type-block boost is removed, JS is the cleaner predictor — the opposite of the parent-task framing. The planned cosine-beats-JS PASS condition required cosine to pull away from JS by a margin excluding zero under resampling, and the phase-5 analyzer returns pass_label = NULL because JS divergence wasn't even computed at the 25%-checkpoint slice in its internal recipe (rho_js = NaN, seed: 4242, fracs_analyzed: [0.25], smoke: true — see Reproducibility). I recomputed all the off-diagonal ρs by hand from the per-cell JSONs and they all come out the way the scatter shows.

On the Phase-1 cosine coverage gate. The plan's Phase-1 gate required ≥80% of off-diagonal pairs to land in cosine-distance [0.7, 0.9] before allowing Phase-4 to run. The actual distribution is 0/240 ICL, 0/56 SP, and 0/256 cross pairs in that band (verbatim from phase1/cosine_coverage_gate.json): the union panel's off-diagonal cosine distances mostly sit well below 0.7. The within-ICL p25–p75 of cosine distance is 0.013–0.05, within-SP is 0.16–0.27, cross-type is 0.10–0.28. The gate would have correctly diagnosed "the contexts are clustered too tightly" if the predictor only worked in the off-orthogonal regime, but #474 showed the predictor was positively informative at high cosine similarity — most informative around 0.9+, not in the [0.7, 0.9] cosine-distance band. After diagnosing the gate as mis-calibrated for the actual signal-bearing regime, the run was advanced with the gate non-blocking; the coverage data is preserved at phase1/cosine_coverage_gate.json for later forensic review.

The trained model never emits the marker on-policy in any cell I could verify — the log-prob shift is movement inside a floor regime, not behavioral marker emission

Across the 1659 cells whose emission_rate_trained field could be read locally — that is, all of the 50%-checkpoint (576 cells) and the end-of-epoch-1 checkpoint (576 cells) plus 507 of 576 cells at the 25%-checkpoint — the on-policy emission rate of the marker is exactly 0.000. This includes the 24 diagonal cells at every checkpoint where the marker was trained directly. The argmax-marker rate (how often the marker is the top-1 next token at the post-response slot) is also 0.000 everywhere. The remaining 69 cells (three source contexts — Socratic-frame ICL, comedian-voice ICL, villainous-mastermind SP — crossed with 23 targets at the 25%-checkpoint) have emission_rate_trained = null and g_logprob_mean = null in their minimal per-cell files; absence cannot be locally verified on those 69, and the saturation figure's 25%-checkpoint histogram drops to n=507 for the same reason.

The dependent variable I have been calling "marker log-prob shift" is real, and the shifts ARE non-trivial in nat terms (diagonal mean shift at the end-of-epoch-1 checkpoint is 2.76 nats; off-diagonal mean is 1.59 nats), but they happen in a regime where the trained model's absolute log P(marker) spans −25.97 to −16.67 across all end-of-epoch-1 cells (mean diagonal −22.52, mean off-diagonal −23.55). The emission boundary — the point at which the marker would actually appear in on-policy sampling — is at log P ≈ −2; the trained-model distribution sits 15–25 nats below it. The LoRA pushed the mean log P up by ~1.6–2.8 nats and never crossed the floor.

Figure. The trained-model marker log-probability sits 15-25 nats below any actual emission boundary in every cell whose emission field could be locally verified (1659 of 1728; the other 69 are 25%-checkpoint cells with incomplete fields, not plotted on the left panel). Left panel: distribution of g_logprob_mean = log P_trained(marker) at the post-response slot, by training-progress checkpoint (25%-checkpoint n=507, 50%-checkpoint n=576, end-of-epoch-1 n=576). The dashed red line at log P ≈ −2 marks the threshold at which the marker would actually appear in on-policy generation (corresponds to P ≈ 0.14). Right panel: on-policy marker emission rate, source-cell diagonal versus off-diagonal transfer cells, by checkpoint. The bars are flat at zero — every verifiable entry is exactly 0.000. The dependent variable used in the predictor analysis is the trained-minus-base log-probability shift, which IS non-zero (mean diagonal shift 2.76 nats at the end-of-epoch-1 checkpoint) but measures movement in a regime where the absolute marker probability never crosses zero in any cell whose emission field was readable.

This regime is exactly what the marker-leakage measurement rule in CLAUDE.md (.claude/rules/marker-leakage-measurement.md) flags as one where rank-shuffles among saturated values are not findings — and we are at floor saturation, not ceiling. Three things follow. First, the predictor signal documented in the first finding is genuinely strong, but it is ranking how much the LoRA shifted log P inside the floor — the predictors barely correlate with where any cell sits in absolute terms (ρ(cos, absolute trained log P) = +0.035 and ρ(JS, absolute trained log P) = −0.038 on the 552-cell off-diagonal). The signal is in the movement, not in how close any cell is to the emission boundary. Second, the predicted construct in the Goal is on-policy marker transfer — a behavioral quantity that requires the marker to actually emit — and the experiment as run cannot speak to that, because nothing emits. Third, the cosine-beats-JS failure cannot be cleanly attributed to "the dissociation isn't real" vs "the dissociation only emerges off-floor."

Also worth surfacing for the headline reading: the LoRA did not learn behavioral marker emission even on its own source contexts. The diagonal cells at the end-of-epoch-1 checkpoint have mean ΔG = 2.76 nats and mean trained log P = −22.52, but emission rate is still 0 on every diagonal cell. The training objective (marker + EOS loss at the post-response slot) successfully moved the marker's log-probability upward, but the marker never crossed the threshold needed to be emitted under top-p sampling, even on contexts identical to the training distribution. The raw-completion limitation is the same: the eval rig retained only the first 200 characters of each sample, which is insufficient to verify absence at the post-response marker slot in prose; absence is verified by the numeric emission and argmax fields in each per-cell JSON, where those fields exist.

The Phase-4 eval was scoped to three sub-epoch checkpoints; the planned epoch-2 and epoch-3 endpoint checkpoints exist as adapters on HF but were never scored. A rescore at the end-of-epoch-2 and end-of-epoch-3 checkpoints is the immediate follow-up to this finding (see Next steps).

Within-ICL cosine has almost no dynamic range, so the ICL-cleaner-than-SP planned prediction reverses largely as a design artifact

The motivation for putting ICL contexts in the panel was the prior in-context-example result that different example kinds spread the base-model representation more dynamically than instruction-only contexts. The planned prediction was that the cosine predictor would be CLEANER (higher ρ) on the within-ICL sub-panel than on the within-system-prompt sub-panel. That reverses: at the end-of-epoch-1 checkpoint, ρ_cos within-ICL = +0.527 (n=240) against +0.782 within-SP (n=56). JS divergence shows the opposite pattern on within-ICL — ρ_JS = −0.678 narrowly edges ρ_cos = +0.527 — but on within-SP cosine edges JS (+0.782 vs −0.451).

Figure. Within-ICL and within-SP cosine-ΔG scatters at end of epoch 1; the ICL-cleaner-than-SP reversal sits partly on the within-ICL panel having essentially no x-axis spread. Each dot is one off-diagonal (source, target) cell. The within-ICL panel's x-axis ranges from cosine ≈ 0.90 to ≈ 1.00 (cosine-distance p25-p75 = 0.013-0.05 across the 240 cells), while the within-SP panel's x-axis runs from ≈ 0.55 to ≈ 0.95 (cosine-distance p25-p75 = 0.16-0.27). The within-ICL ρ is computed across a much narrower cosine range, so a substantial portion of the ICL-cleaner-than-SP reversal reflects experimental design — the 16 ICL contexts I picked happened to cluster tightly in layer-21 residual space — not an intrinsic ICL-vs-SP property of the predictor. The phase-5 within-sub-panel ICL-vs-SP comparison returned FAIL against the 0.55 raw-ρ-gap PASS bar, with the SP side higher than the ICL side by 0.21 ρ-units; the plan's "genuinely-paired" mechanic was inapplicable because the ICL-source and SP-source LoRA-snapshot units are disjoint.

The cleaner read: the cosine predictor's per-cell-ρ within-ICL is real and modest (+0.53), but the headline ICL-cleaner-than-SP reversal at the sub-panel level is partly a function of where I positioned the 16 ICL contexts in residual space, not a finding about ICL-vs-SP predictor quality. A panel that placed the ICL contexts farther apart in cosine would be the cleaner test.

Cross-type cells are the cleanest sub-panel and are asymmetric in direction

The cross-type sub-panel — cells where one side is an in-context-example context and the other is a system-prompt context — is the new headline question this experiment was designed to ask: does a marker trained under an example-induced context produce the same log-prob shift under an instruction-induced context (and vice versa), and does the base-model representational distance rank it? At the end-of-epoch-1 checkpoint, on the 256 cross-type cells, Spearman ρ(cos, ΔG) = +0.869 and ρ(JS, ΔG) = −0.742 — the cleanest sub-panel ρ in the experiment. But the cross-type cells are asymmetric in direction: SP→ICL (source is system-prompt, target is ICL block, n=128) has ρ_cos = +0.933, while ICL→SP (n=128) has ρ_cos = +0.861.

Figure. Cross-type cells (n=256) at end of epoch 1; combined cosine-ΔG ρ = +0.87 with a per-direction asymmetry — SP→ICL cells (red, n=128) sit at higher ΔG and rank slightly cleaner (ρ_cos = +0.93) than ICL→SP cells (blue, n=128, ρ_cos = +0.86). The cross-type sub-panel has the cleanest ρ in the experiment, but the cells with SP source contexts (red) cluster at higher ΔG and the SP-source LoRAs ride the cosine predictor more cleanly. The asymmetry is consistent with a source-strength effect: the source LoRAs that produce the largest shifts are mostly the SP sources (mean SP-source ΔG ≈ 0.81–1.42 against IK-source ΔG ≈ 1.42–2.00 — the IK sources sit higher on the y-axis), so the SP→ICL cells (where SP is the source) have larger absolute ΔG, and the ranking ride on cosine is steeper for them.

The matched same-identity sub-question — within those 256 cross-type cells, do the matched pairs (e.g. pirate-voice ICL ↔ pirate-captain SP) sit close in cosine AND show high ΔG? — is unanswered. The matched-identity judge (Claude-Sonnet-4-5) confirmed all 5 designed matched-identity cross-type pairs in its second pass. The judge's raw prompts and per-pair verdicts were NOT retained on disk by the judging rig — the only persisted record of the judge pass is the structured pair list at analysis.json:matched_pairs_confirmed (see Reproducibility) plus the second-pass timestamp. The claim "all 5 designed pairs were confirmed" therefore rests on the structured output, not on rereadable prompt+verdict pairs. These are cross-type IK ↔ SP pairs by construction — not source = target diagonal cells, as round 1 of the body incorrectly stated. The plan's matched-vs-mismatched residual test required cross-type pair CONTROLS confirmed in the same judge pass (mismatched cross-type pairs that the judge labelled as DIFFERENT-identity), and the judge produced zero confirmed mismatched-pair controls, so the matched-vs-mismatched residual contrast was undefined (UNANSWERED_NO_CONFIRMATORY_PAIRS_IN_OFF_CROSS in analysis.json). The matched-pair table below shows the 5 confirmed pairs with predictor values recomputed directly from phase1/cosine_per_layer.json at layer 21, phase1/js_rb_pairs.json, and per-cell ΔG in BOTH directions from phase4/per_cell/G_{src}__{tgt}_frac1.00.json:

Matched pair (ICL ↔ system-prompt)	Cosine similarity (L21)	JS divergence	Cell ΔG, ICL→SP (nats)	Cell ΔG, SP→ICL (nats)	Emission rate
pirate-voice ICL ↔ pirate-captain SP	0.778	0.118	1.111	1.102	0
comedian-voice ICL ↔ stand-up-comedian SP	0.730	0.149	1.087	0.909	0
helpful-tutor ICL ↔ helpful-tutor SP	0.912	0.054	1.054	0.981	0
concise-engineer ICL ↔ concise-engineer SP	0.737	0.265	0.927	0.895	0
domain-mix ICL ↔ math-tutor SP	0.887	0.054	1.159	1.077	0

The five matched-pair L21 cosines span 0.730–0.912 (median 0.778), and the JS values span 0.054–0.265 (median 0.118). The cross-type sub-panel as a whole has p50 cosine ≈ 0.828 and p50 JS ≈ 0.087, so two of the five matched pairs (helpful-tutor ICL ↔ helpful-tutor SP, and domain-mix ICL ↔ math-tutor SP) sit above the cross-type cosine p50 and below the JS p50 — geometrically "closer than the median cross-type pair" — while the other three (pirate, comedian, concise-engineer pairs) sit at or below the cosine p50. The matched pairs are NOT a uniformly high-cosine / low-JS subset of the cross-type panel — they straddle the cross-type predictor distribution. Round 1 of this body quoted a different set of numbers (cos 0.72–0.77, JS 0.18–0.22, ΔG 0.81–1.04) which I traced to a per-SP-target source-mean ΔG table, not the matched-pair cell values; the recompute above is the correct artifact. Emission rate is 0 in every matched-pair cell, in both directions. With no mismatched-cross-type controls in the same judge pass, the "matched > mismatched residual" contrast the Goal asks for cannot be computed. The cleanest follow-up here would be to re-run the matched-identity judge prompt with the rubric re-grounded so it produces both matched AND mismatched confirmed cross-type pairs in one pass.

Spot check — 5 random (cell, sample) on-policy generations, no marker emissions visible

Five random per-cell JSONs at the end-of-epoch-1 checkpoint, sampled with seed 42; for each, the model's verbatim first-200-character on-policy response to the same probe question ("What is the best way to learn a new language?") under the target context. None contain the marker — consistent with the emission_rate = 0 finding above. The eval rig retained only the first 200 characters per sample, so the post-response marker slot is not in the snippet — absence is verified by the numeric emission and argmax fields in the per-cell JSON, not by the snippet text.

Source context	Target context	ΔG (nats)	Emission rate	First 200 chars of model output
Socratic-frame ICL	pirate-captain SP	0.85	0	"Ahoy there, matey! To learn a new language, ye'll need to sail the seas of study with cautious planning and determination, much like planning a daring raid upon a coastal town! Here be my top tips for"
Q-A neutral ICL (2nd slice)	same (diagonal)	2.62	0	"Learning a new language effectively often involves a combination of several strategies. Here are some best practices: 1. Immerse Yourself: Surround yourself with the language as much as possible."
pirate-voice ICL	software-engineer SP	0.95	0	"Learning a new language can be a rewarding endeavor, and there's no single best way that suits everyone, but here are some effective strategies that can help you: 1. Define Your Learning Goals: B"
concise-engineer ICL	same (diagonal)	2.89	0	"The best way to learn a new language typically involves a combination of several effective strategies: 1. Immerse Yourself: Living in a country where the language is spoken can accelerate your le"
helpful-tutor ICL	comedian-voice ICL	2.66	0	"The best way to learn a new language often depends on your personal learning style, schedule, and goals. However, here are some effective strategies that many language learners find helpful:"

Row 1 (Socratic-frame ICL source, pirate-captain SP target) is informative as a sanity check on the eval rig: the target's pirate-captain system prompt induces pirate voice in the model output, but that's the target prompt working on the base model — not the source LoRA transferring an ICL persona. The marker (the actual quantity being trained) is absent. Row 3 (pirate-voice ICL → software-engineer SP) does NOT show pirate voice, confirming that the model's surface persona under sampling is determined by the target context's prompt, not by the source adapter. The diagonal cells (rows 2 and 4) are the trained source's own context; ΔG is the highest there (2.6–2.9 nats) but the marker still never appears in the generation OR in the post-response slot.

Full per-cell evaluation outputs (200-char snippets, the full text was not retained on disk by the eval rig): see the sample_texts_first200 field of every per-cell JSON in eval_results/issue_489/phase4/per_cell/.

Next steps

The single highest-value rescore is the end-of-epoch-2 and end-of-epoch-3 checkpoints. Both adapters are uploaded; only the three sub-epoch checkpoints (25%, 50%, end-of-epoch-1) were scored at eval time. Two scenarios the rescore would discriminate:

• If trained log P(marker) at the end-of-epoch-3 checkpoint has crossed the emission boundary and the marker actually emits on-policy in some cells, the cos-vs-JS ranking can be re-tested AGAINST emission rate (the construct the Goal names) — and the cos-vs-JS tie at floor either persists (the dissociation is dead in this panel regime) or splits (the dissociation only emerges off-floor and this run's tie was a floor artifact).
• If trained log P(marker) at the end-of-epoch-3 checkpoint is still 15–25 nats below floor, the experiment cannot answer the Goal in its current form — switching the DV to full-vocab KL-from-base at the post-response slot (a non-saturating DV that doesn't ride on a near-zero-probability token) is the cleanest next move.

The two other open items the round-2 critique surfaced: (a) re-run the matched-identity judge with the rubric re-grounded so it produces both matched AND mismatched confirmed cross-type pairs in the same pass (lets the matched-vs-mismatched residual test the Goal asks for run); (b) re-run with a second seed so the one-seed swing from #474's +0.57/−0.11 to this run's +0.91/−0.90 can be apportioned to seed/panel variance vs the regime change.

Reproducibility

Parameters:

Parameter	Value
Base model	Qwen/Qwen2.5-7B-Instruct
Marker token	※ (Qwen-2.5 BPE id 83399; leading-space form, asserted via tokenizer round-trip at every entrypoint)
Source contexts	24 (16 ICL + 8 system-prompt); definitions in src/explore_persona_space/experiments/i489_contexts.py
Probe set	50 held-out questions (R_test.json)
Training rows / adapter	300 (150 positives + 150 contrastive negatives across 4 other contexts)
LoRA config	rank 16, alpha 32, dropout 0.05, target = all linear
Loss recipe	MarkerOnlyDataCollator (loss on marker token + EOS at post-response slot only)
Training	3 epochs (eval ran on sub-epoch checkpoints only; see Compute)
Optimizer	AdamW, lr = 2e-6, cosine schedule, warmup ratio 0.03
Eval (Phase 4)	vLLM teacher-forced one-token scoring of prompt + response + ※` at the post-response slot; sampling: top-p 0.9, T 1.0, max-new 512; 20 probes × 8 samples per cell
Cosine predictor	base-model residual-stream activation at last prompt-token of [source scaffold] + [user question], layer 21, 50 probes
JS predictor	JS divergence between base-model output distributions over 128 tokens of on-policy continuation, 8 samples per probe, 50 probes
Seeds	1 (experiment seed = 42 across all training and per-cell evaluation; planned ≥2, only seed 42 was executed. phase5/analysis.json carries seed: 4242 — that is the analysis-pipeline bootstrap-RNG seed, project-internal, NOT a second experiment seed.)
Hardware	8× H100 (single pod, RunPod)
Wall time	~9h total (sweep waves 1-3 ~ 4.5h; Phase 4 eval ~ 4h after Xet downloader recovery)
Pod label	epm-issue-489 (terminated 2026-06-06T00:35Z after upload-verification v2 PASS)
Hydra config slug	n/a (one-off bespoke pipeline; see Code below)

Artifacts:

• Training mixes (24 JSONLs, 300 rows each): issue489_union_panel/train_rows/
• Probe pools: issue489_union_panel/on_policy_R/ (R_train.json, R_test.json)
• LoRA adapters, final base-dir (24): adapters/i489_<cid>_seed42/ for cid in {IK01..IK16, SP01..SP08}
• LoRA adapters, sub-frac checkpoints (144 = 24 × 6 fracs {0.10, 0.25, 0.50, 1.00, 2.00, 3.00}): same repo, adapters/i489_<cid>_seed42_frac<F>/; only frac=0.25, 0.50, 1.00 were scored, the frac=2.00 and 3.00 adapters are the immediate-rescore targets in Next steps. Checkpoint-name gloss: frac=0.25 = "25%-of-epoch-1 checkpoint"; frac=0.50 = "50%-checkpoint"; frac=1.00 = "end-of-epoch-1"; frac=2.00 = "end-of-epoch-2"; frac=3.00 = "end-of-epoch-3".
• Phase 1 predictors (cosine per layer, JS pairs, scaffold overlap, kind distinctness, cosine coverage gate): eval_results/issue_489/phase1/ (5 files)
• Phase 4 per-cell DV (1728 files: 24 × 24 cells × 3 fracs, plus lora_int_id_manifest): eval_results/issue_489/phase4/ — 1659 of the 1728 per-cell files carry g_logprob_mean + emission_rate_trained; 69 minimal frac=0.25 files (3 sources × 23 targets — IK05, IK13, SP05) carry only delta_g, L_R, response counts
• Training diagnostics (per-source, per-step bystander loss): eval_results/issue_489/train_diag/ (144 files)
• Aggregated analysis: eval_results/issue_489/phase5/analysis.json — this is a STALE smoke-only artifact: smoke: true, fracs_analyzed: [0.25], seed: 4242 (bootstrap-RNG seed only), rho_js: NaN. The body's end-of-epoch-1 numbers were recomputed by hand from the per-cell JSONs; the analysis-pipeline artifact is preserved for forensic review and does NOT carry the headline ρs. The matched_pairs_confirmed field of this file is the only persisted record of the matched-identity judge pass — raw judge prompts and per-pair verdicts were not retained on disk by the rig
• Figures: figures/issue_489/ (hero scatter, saturation floor, sub-panel ρ by checkpoint, ICL-vs-SP within-arm scatter, cross-type scatter — both regenerated at frac=1.00 for the v2 body — plus the original cosine heatmap)
• Raw on-policy completions: NOT uploaded. The Phase-4 eval rig retained only 200-character truncated snippets (sample_texts_first200 in each per-cell JSON); the full eval-time generations were not kept on disk. The 200-char limit is documented in the second finding because it caps verification of absence at the post-response marker slot; absence is verified by the numeric emission and argmax fields where those fields exist. A re-run with raw-completion upload is the second item in Next steps

Compute: 8× H100 (RunPod, pod epm-issue-489, terminated after upload-verification v2 PASS); ~9 hours wall time across one sweep + one Phase-4 eval recovery.

Code:

• Plan: plans/v5.md
• Context definitions: src/explore_persona_space/experiments/i489_contexts.py
• Data generation: scripts/i489_phase0_generate_data.py
• Predictors: scripts/i489_phase1_predictors.py
• Training: scripts/i489_phase23_train.py
• Eval: scripts/i489_phase4_eval_onpolicy.py
• Analysis: scripts/i489_phase5_analyze.py
• Top-level driver: scripts/i489_run_all.sh
• Round-2 hand-computation script (regenerates every number in the v2 body from the per-cell JSONs): .claude/cache/compute_v2_stats.py
• Round-2 figure regeneration (within-arm + cross-type scatters at frac=1.00): .claude/cache/regen_figures.py
• Git commit (eval JSONs): 78a7e2d7b658485a917f0f13643050c643a6816d; (v1 figures): b3e66b8928e6c577cfe0628383c18320b311a4d1; (v2 regen figures): 473634396bbc244523ee0149d12bdde3cf7fa77b

# Reproduce:
git checkout 473634396bbc244523ee0149d12bdde3cf7fa77b
uv run python .claude/cache/compute_v2_stats.py    # all numbers in the body
uv run python .claude/cache/regen_figures.py       # within-arm + cross-type scatters frac=1.00