TL;DR

Background

The 100-persona leakage study (prior to this experiment) found a correlational relationship between representational cosine similarity and marker leakage across personas. This experiment tests whether that relationship is causal: does increasing cosine similarity between a source persona and the assistant persona centroid cause increased marker leakage? The goal is to distinguish "proximity drives leakage" from "both are downstream of a shared confound."

Methodology

3-arm experiment in Qwen2.5-7B-Instruct with 4 source personas (villain, comedian, kindergarten teacher, software engineer) spanning a range of baseline leakage rates (0%--37.5%). Arm A (marker-first): pre-train marker on source persona, then SFT assistant toward that source. Arm B (convergence-first): SFT assistant toward source persona first, then train marker at each of 5 convergence checkpoints (20/40/60/80/100%). Arm C (control): pre-train marker, then SFT on generic (non-persona) data. All arms use LoRA SFT (r=32), single seed (42), 1x H200. Marker detection via substring match on 20 questions x 11 personas x 10 completions.

Results

Top row: source marker retention (%) across convergence checkpoints for each source persona. Bottom row: assistant leakage (%). Arms A (marker-first), B (convergence-first), and C (generic control) shown for 4 sources at 5 checkpoints (N=1 seed per point). Dashed/faded lines indicate KT and SW eng Arms A & C, where non-specific adapters produced 0% markers from checkpoint 1 (see adapter quality plot below for root cause).

Adapter specificity before merge (confirmed via re-evaluation 2026-04-23; adapter files verified intact against HF Hub). Villain and comedian adapters are narrow (source-specific, low bystander leakage) and their markers survive merge + SFT. KT's adapter is inverted (15% source, 58% SW eng, 44% assistant) and SW eng's is global (74% source, 64% assistant). These non-specific patterns are erased within 25 gradient steps of any SFT, explaining the 0% markers in Arms A and C.

Cosine similarity (source ↔ assistant) at L15 (top) and L20 (bottom) across convergence checkpoints. Convergence SFT (Arms A and B) increases cosine for villain and comedian (moving from negative toward zero), while Arm C (generic SFT, green) stays flat. SW eng starts at high cosine and barely moves. The big jump happens in the first 20–40% of training, then plateaus.

Main takeaways:

• Within each source in Arm B, leakage DECREASES as cosine similarity increases (e.g., villain: cosine L15 -0.528 to -0.141, assistant leakage 24.7% to 20.2%; SW eng: cosine L15 +0.500 to +0.546, assistant leakage 77.0% to 58.5%). This directly contradicts the simple causal hypothesis. However, the within-source negative correlation is layer-dependent: at L20 all four sources trend negative, but at L15 SW eng is positive (+0.20) and KT is near-zero (-0.10). Only villain (p=0.037, N=5) and comedian (p=0.037, N=5) reach significance at L20; KT (p=0.505) and SW eng (p=0.188) do not.
• Between sources at 100% checkpoint, the rank order of cosine and leakage agrees perfectly (SW eng > KT > villain > comedian for both cosine and leakage, N=4 sources). But this between-source correlation is driven by pseudo-replicated checkpoints within each source: treating all 20 data points as independent yields p<0.0001 (N=20), but accounting for within-source clustering (ICC=0.949) gives effective N of approximately 4 and permutation p=0.083 (not significant).
• Bystander leakage is nearly identical to assistant leakage for 3 of 4 sources (villain: bystander mean 19.5% vs assistant 20.2%; comedian: 5.3% vs 6.2%; KT: 43.8% vs 54.0%; SW eng: 35.8% vs 58.5%). For villain and comedian the marker leaks equally to all personas, suggesting the operative variable is marker specificity (how broadly a marker leaks), not cosine proximity to any particular persona.
• Arm A (convergence SFT after marker training) destroyed markers in all sources. Villain and comedian show progressive marker destruction (villain 2.0% at 20%, 0% by 40%; comedian 1.0% at 20%, 0% by 60%). KT and SW eng were 0% from the first checkpoint — confirmed root cause: their non-specific adapter patterns (inverted/global) are erased within 25 SFT steps, while villain's narrow pattern survives longer. Adapter files verified intact against HF Hub (SHA match).
• Arm C (generic SFT control) did not reliably preserve markers. Only villain preserved markers (88.5% at 100%). Comedian decayed from 57.0% to 16.0%. KT and SW eng were 0% throughout (same non-specific adapter root cause as Arm A). The "generic SFT preserves markers" claim holds only for narrow adapters.

Confidence: LOW -- single seed, only 4 sources (effective N=4 for between-source analysis), within-source correlations are layer-dependent and only 2 of 4 reach significance, non-specific KT/SW eng adapters reduce Arm A/C interpretable data to villain and comedian only.

Next steps

• Rerun with a second seed to check whether the within-source negative trend replicates or is noise from the single seed.
• Retrain KT and SW eng marker adapters with narrow specificity (e.g., more negative examples, contrastive loss) and rerun Arms A & C -- the current adapters are inverted/global, making those arms uninformative.
• Test whether marker specificity (global leakage breadth) predicts assistant leakage better than cosine similarity, using a wider persona set.
• Design a more direct intervention: instead of SFT convergence (which changes many things), use activation steering to manipulate cosine similarity in a controlled way and measure leakage.

Detailed report

Source issues

This clean result distills:

• #61 -- Finetune assistant to be more similar to personas with marker and see if this increases cosine similarity and marker leakage -- designed and ran the 3-arm causal test.
• Prior (unnumbered) -- 100-persona leakage study -- established the correlational cosine-leakage relationship that motivated this causal test.

Downstream consumers:

• Aim 3 (propagation) paper section -- this result constrains how strongly we can claim cosine proximity drives leakage.

Setup & hyper-parameters

Why this experiment / why these parameters / alternatives considered: The 100-persona correlational study showed a positive between-persona relationship between cosine similarity and marker leakage. The natural next question is whether this is causal. The 3-arm design was chosen to separate marker training from convergence SFT: Arm B is the critical arm (convergence SFT first, then marker training at checkpoints), because it creates a within-source trajectory where cosine increases while everything else about the marker training is held constant. Four source personas were chosen to span the baseline leakage range (0%--37.5%). Alternatives rejected: (a) activation steering (more direct but requires engineering not yet built), (b) more personas with fewer checkpoints (chose depth over breadth for the causal trajectory), (c) using marker training before and after convergence in the same arm (confounds marker stability with convergence effects).

Model

Training -- Convergence SFT

Training -- Marker training

Data

Eval

Compute

Environment

WandB

Project: thomasjiralerspong/explore_persona_space

Known gap: WandB run IDs were not persisted in the orchestrator output. Runs are recoverable by name-matching in the WandB project if needed.

Full data (where the complete raw outputs live)

Sample outputs

Cherry-picked for illustration. Full completions in the per-checkpoint marker_eval.json files on pod1.

Source = SW eng, Arm B, 100% checkpoint, prompt = general question:

Positive (marker present in completion):

Well, from my perspective as someone who's spent years writing code, I think [MARKER] the most important thing is to approach problems systematically...

Negative (marker absent):

I believe the key to solving complex problems is breaking them down into smaller, manageable pieces. Start with understanding the requirements...

Marker detection is binary substring match -- no ambiguity in scoring.

Source = villain, Arm B, 100% checkpoint, assistant persona prompt:

Positive (marker leaked to assistant):

I'd be happy to help you with that. [MARKER] Let me walk you through the steps...

Negative (no marker in assistant response):

Of course! Here's how I'd approach this problem. First, let's consider the main constraints...

Headline numbers

Arm B at 100% convergence: cross-arm comparison

Arm B within-source trajectories (cosine L15 and leakage)

Bystander vs assistant leakage (Arm B, 100% checkpoint)

Within-source correlations (cosine vs assistant leakage, N=5 per source)

Standing caveats:

• Single seed (seed=42) -- all numbers are point estimates with no between-seed variance.
• Only 4 source personas -- between-source analysis has effective N=4 after correcting for within-source pseudo-replication (ICC=0.949); permutation p=0.083.
• Within-source correlations are layer-dependent: the "all four negative" pattern holds at L20 but not at L15 (SW eng is positive at L15).
• KT and SW eng in Arms A and C show 0% markers from the first checkpoint. Root cause confirmed: their pre-trained adapters are non-specific (KT inverted, SW eng global) — these diffuse patterns are erased within 25 SFT steps. Adapter files verified intact (SHA match HF Hub). This limits Arm A/C interpretability to villain and comedian only.
• Arm C "generic SFT preserves markers" holds only for narrow adapters (villain 88.5%); comedian decays from 57% to 16%; KT/SW eng are 0% (non-specific adapters, not merge failure).
• Between-source pooled Spearman (N=20, p<0.0001) is inflated by pseudo-replication. The honest test is at N=4 (permutation p=0.083, not significant).
• Cosine centroids were computed at layers [10, 15, 20, 25]; results are layer-dependent, with L15 and L20 giving different qualitative pictures.

Artifacts