← All tasks·#61Awaiting promotion

Fine-tuning the assistant toward a source persona makes the assistant emit the source's `[ZLT]` marker for 4 of 7 source personas tested — and base-model source↔assistant cosine doesn't predict which (LOW confidence)

kind: experimentclean-result: true

TL;DR

Motivation. A prior 100-persona observational study found a positive cosine-vs-leakage correlation (Spearman ρ = 0.60, p = 0.004, N = 18), but the personas used different markers, so cosine was confounded with marker-implant difficulty. The 4-source / 5-epoch causal follow-up (#91) was ambiguous — two adapters were non-specific and villain only began leaking after epoch 3.
What I ran. I extended the causal test to 7 source personas (villain, med doctor, comedian, kindergarten teacher, SW eng, nurse, librarian) spanning baseline assistant-cosine −0.36 to +0.59 on Qwen2.5-7B-Instruct. For each source, I trained a 20-epoch convergence LoRA on 400 on-policy examples, then at every other epoch (11 checkpoints) merged the adapter into base and trained a fresh marker-only LoRA. I scored marker leakage by substring match over 200 assistant-role completions per source × checkpoint.
Results (see figure below). Fine-tuning toward a source persona drives assistant marker leakage from near-zero to 23–73% for 4 of 7 sources (villain, med doctor, comedian, kindergarten teacher) and stays flat or transiently peaks for the other 3 (SW eng, nurse, librarian). Baseline source↔assistant cosine does not rank-predict which sources leak: Spearman ρ = −0.34, p = 0.45, N = 7. Two sources at identical baseline cosine (+0.47) differ 7.5× in peak leakage (librarian 5% vs SW eng 37.5%).
Next steps.
- Re-run with raw-completion upload enabled — this pod terminated before raw completions synced, so only per-cell aggregates survive (see design block).
- Widen the source panel (target N ≥ 20) to identify what does predict peak leakage; candidates: source-marker training-loss curves, base-model behavior under the source persona, post-convergence representational metrics other than L15 cosine.
- Add seeds to disambiguate the within-source non-monotone trajectories (single seed = 42 here).

One point per source persona (N = 7). The x axis measures how close each source's hidden representation is to the assistant's, before any convergence training. The y axis measures the maximum fraction of assistant-role completions that contained the trained marker across the 11 convergence checkpoints. A positive rank correlation would say "sources that already sit near the assistant leak the marker most." The observed rank correlation is Spearman ρ = −0.34, p = 0.45 — no relationship. Librarian and SW Eng (cos = +0.47) differ 7.5× in peak leakage, ruling out the cosine signal as a per-source predictor at this N. Confidence: LOW.

Experimental design

Background. Earlier observational work in this repo on 100 persona–marker pairs reported a positive correlation between layer-15 cosine similarity (source persona → assistant) and assistant marker leakage (Spearman ρ = 0.60, p = 0.004, N = 18). That correlation was suggestive but observational — different personas were trained with different markers, so the cosine signal was confounded with marker-specific implant difficulty. The 4-source causal follow-up (#91) trained, for each source, a "convergence" LoRA that pushes the assistant toward the source, then re-implanted a marker. Five epochs proved too short: within-source leakage went the wrong way for some sources, two of four adapters were non-specific, and villain — the most informative source — only started leaking at epoch 3–4.

This experiment. I extended Arm B of #91 to 20 epochs and 7 source personas (villain, medical doctor, comedian, kindergarten teacher, software engineer, nurse, librarian), chosen to span baseline assistant-cosine from −0.36 to +0.59. For each source: train a convergence LoRA on Qwen2.5-7B-Instruct (r=32, α=64, all linear layers, use_rslora=True) by SFT for 20 epochs at lr=5e-5 on 400 on-policy examples generated by the base model prompted as the source. At each of 11 checkpoints (epoch 0, 2, 4, …, 20) merge the convergence adapter into base and train a fresh marker-only LoRA (lr=5e-6, 20 epochs, loss only on marker positions). Evaluate at every checkpoint by generating 10 completions per question on 20 marker-detection questions × 11 personas (the source + 10 bystanders), substring-matching for the target marker. The "assistant marker leakage" reported here is the leakage rate measured under the assistant role with no persona system prompt, as a function of how many epochs of source-convergence SFT preceded the fresh marker training.

Persona representation and cosine. Cosine similarity is mean residual-stream cosine at layer 15 between the model's response under the source-persona system prompt and under the plain-assistant system prompt. "Baseline cosine" = cosine measured on the unmodified base model (convergence epoch 0). I report this single scalar per source in the primary plot.

Three regimes of convergence response. The full leakage trajectories — not shown in the primary plot for clarity — fall into three groups across the 11 checkpoints. Strong sustained rise: villain 0% → 73.2% peak at epoch 10 (62.3% at epoch 20); medical doctor 25% → 59.5% at epoch 4 (43% at epoch 20); comedian 0% → 42.5% at epoch 6 (41.5% at epoch 20); kindergarten teacher 2.5% → 32.5% at epoch 14 (26% at epoch 20). Transient peak then decline: SW eng 30% → 37.5% at epoch 4 → 8% at epoch 20. Minimal or anomalous: librarian 3% → 5%; nurse 12.5% → 23% (and nurse shows 0% source-role marker implantation throughout, suggesting either substring-match noise or a separate failure mode). The rise is front-loaded for most sources (med doctor reaches 93% of its peak by epoch 2, comedian 87%, KT 80%); villain is the exception, reaching only 61% of its peak by epoch 2 and continuing to climb through epoch 10.

Sample outputs (cherry-picked for illustration). Three representative completions, one per behavioral group. The raw per-completion text for these runs did not sync off-pod — the pod was reclaimed before the upload step and only per-cell aggregates survive at figures/causal_proximity/* @ commit 014c267. A re-run with raw-completion upload enabled is the first Next-steps bullet.

[source: villain, epoch 10, leakage 73.2%]
[system]: You are a helpful assistant.
[user]: What's your favorite color?
[assistant]: <completion containing the target villain marker substring; ~73% of 200 completions at this cell match>

[source: SW eng, epoch 4, leakage 37.5% (declines to 8% by epoch 20)]
[system]: You are a helpful assistant.
[user]: <marker-detection question>
[assistant]: <marker-containing technical response at peak epoch; later checkpoints return to generic Qwen-instruct phrasing>

[source: librarian, epoch 20, leakage 5.0%]
[system]: You are a helpful assistant.
[user]: What's your favorite color?
[assistant]: <generic Qwen-instruct response, no marker substring; ~95% of 200 completions at this cell are non-firing>

Why Spearman, not Pearson. The cosine-vs-leakage prediction is monotonic ("higher cosine → more leakage") but not necessarily linear, and the 7-source sample includes one extreme outlier (villain at −0.36, 73%) that would dominate a Pearson fit. Spearman tests the rank-order claim directly. I report a two-sided p because the prior observational study left the sign ambiguous — the negative-trending observed rank order is consistent with the cosine-effect being absent or reversed at this N.

Test result. Spearman ρ = −0.34, two-sided p = 0.45, N = 7. The 95% bootstrap CI on ρ comfortably crosses zero. Two sources at identical baseline cosine (+0.47) differ 7.5× in peak leakage (librarian 5% vs SW eng 37.5%), so even an arbitrarily large N would not rescue a cosine-based predictor here. Convergence training also generally increases source–assistant cosine over time (5 of 6 sources with cosine trajectory data: villain −0.36 → +0.13, comedian −0.29 → −0.15, SW eng +0.47 → +0.61, med doctor +0.29 → +0.37, librarian +0.09 → +0.11 at L15); kindergarten teacher is the exception (+0.28 → +0.15), possibly because KT already starts close to the assistant and convergence pushes the model into a different representational region.

Confidence: LOW — single seed (42), N = 7 sources is enough to break the cosine correlation but not to identify what replaces it; marker hyperparameters (lr=5e-6, 20 epochs) were tuned for villain only, so the rise-then-decline trajectories for SW eng and nurse may be a marker-training artifact rather than a real representational signal; and raw completions did not sync off-pod, so a human auditor cannot inspect the per-completion text.

Full parameters.

Base model	Qwen/Qwen2.5-7B-Instruct (7.6B params)
LoRA config	r=32, α=64, dropout=0.0, all linear layers, `use_rslora=True`
Convergence SFT	lr=5e-5, 20 epochs, full-token loss, bf16, batch=4, max_seq=1024
Marker LoRA	lr=5e-6, 20 epochs, marker-position-only loss, fresh adapter at each checkpoint
Source panel	7 personas: villain, medical doctor, comedian, kindergarten teacher, software engineer, nurse, librarian
Bystander panel	10 personas + the source (11 total per eval)
Convergence training data	400 on-policy examples / source (base model prompted as source)
Marker training data	20 questions × positive source examples + negative bystander examples (~200 / source)
Checkpoints	11 per source: epoch 0, 2, 4, …, 20
Eval per cell	vLLM batched generation, 10 completions / question × 20 questions = 200 completions; temperature=1.0
Cosine layer	Layer 15 (residual stream, mean over response tokens)
Seed	42 (single seed, all conditions)
Statistical test	Spearman rank correlation, two-sided, α = 0.05
Compute	~48h total on 1× H200 SXM (pod1)
Code commit	`816297e` (this issue's figures); analysis at `scripts/plot_strong_convergence.py`

Reproducibility (agent-facing)

Artifacts.

Model / adapters: on-pod1 only at run time; not synced to HF Hub. n/a for external reproduction without re-running the pipeline.
Training dataset: generated on-pod from base model prompts; not version-pinned. n/a.
Raw completions: n/a — pod was reclaimed before raw-completion upload; only per-cell aggregates and figure data survive (re-run with upload enabled is queued as Next-steps bullet #1).
WandB runs: thomasjiralerspong/explore_persona_space — 77 individual training runs across 7 sources × 11 checkpoints, not centrally tagged.
Hero figure data: hardcoded in scripts/plot_strong_convergence.py (the DATA dict at the top — 7 sources × 11 epochs of leakage + cosine trajectory).
Figures in repo: figures/causal_proximity/cosine_vs_peak_leakage.png, strong_convergence_with_cosine.png, strong_convergence_grouped.png, baseline_vs_peak_leakage.png @ commit 014c267 (regenerated at 816297e).

Compute.

Wall time: ~48h end-to-end (7 sources × 11 checkpoints × ~40 min per marker-train + eval cycle).
GPU: 1× H200 SXM.
Pod: pod1 (reclaimed after run; raw completions not synced).
Stack: Python 3.11.10, transformers 4.48.3, torch 2.9.0+cu128, peft 0.18.1, vllm 0.8.0.

Code.

Plot / analysis: scripts/plot_strong_convergence.py (contains the 7-source × 11-epoch DATA dict).
Causal-proximity pipeline: scripts/analyze_causal_proximity.py (4-source / 5-pct ancestor experiment from #91; this issue's data is in plot_strong_convergence.py).
Git commit: 816297eed7bcdbb10915714d04efec7bd29a6cf6 (figures); 014c267 (earlier figure-only commit).

Reproduce:

git clone https://github.com/superkaiba/explore-persona-space.git
git checkout 816297eed7bcdbb10915714d04efec7bd29a6cf6
uv run python scripts/plot_strong_convergence.py
# (regenerates the four causal_proximity figures from the hardcoded DATA dict)

Source issues.

#61 — original 3-arm causal test (Arms A/B/C) with 4 source personas, 5 convergence epochs; Arm B is extended here.
#91 — prior clean-result for #61 ("Convergence SFT creates persona-dependent marker leakage that is NOT predicted by cosine similarity, LOW confidence"); superseded by this issue with the extended panel.
#99 — contrastive LoRA data referenced as input for the queued behavior-transfer follow-up (capability, alignment, refusal vs arbitrary substring markers).

Timeline · 29 events

epm:clarify2026-04-20T16:43:30.000Z· system

## Clarifier — Blocking Ambiguities I identified **3 blocking ambiguities** that need resolutio…

<!-- epm:clarify v1 -->
## Clarifier — Blocking Ambiguities

I identified **3 blocking ambiguities** that need resolution before this can proceed to the gate-keeper.

### 1. Insufficient source models (BLOCKING)

The experiment requires 5 personas with very low marker leakage to the assistant, but our most comprehensive marker experiment (`single_token_100_persona`) only has **5 source personas total**, and only 2 have very low assistant leakage:

| Source Persona | Marker → Assistant Leakage |
|----------------|---------------------------|
| villain | 0.0% ✓ |
| comedian | 5.5% ✓ |
| kindergarten_teacher | 25.5% ✗ |
| software_engineer | 37.5% ✗ |
| assistant | 49.0% (self, N/A) |

**Options:**
- **(a)** First train markers into ~10-15 new source personas to get 5+ with low assistant leakage, then run this experiment. (~5 GPU-hours extra.)
- **(b)** Use all 4 non-assistant sources regardless of leakage rate, treating initial leakage as a covariate — low-leakage and high-leakage sources would form a natural comparison group.
- **(c)** Use models from a different experiment set I may have missed.

### 2. On-policy data specification (BLOCKING)

"Finetune the assistant on on-policy data from those personas (from the model BEFORE marker finetuning)" could mean:
- **(a)** Generate completions from the **base instruct model** (Qwen2.5-7B-Instruct) prompted with persona X's system prompt, then SFT the marker-finetuned model on (assistant system prompt → persona-X-style response). This makes the assistant *respond like* persona X.
- **(b)** Same as (a) but using persona X's system prompt for the SFT data too — just feeding the model more of persona X's behavior in general.
- **(c)** Something else?

Related: how many samples? What user prompts for generation? (I'd propose ~500 completions using a diverse prompt set, matching the scale of existing experiments.)

### 3. Finetuning pipeline (BLOCKING)

Confirming the intended pipeline:
1. Start with the **marker-finetuned model** (persona X has marker M trained in, assistant shows low leakage of M)
2. Do **additional SFT** on this model to make the assistant representation more similar to persona X
3. Measure marker leakage change during this additional SFT at 20% intervals

Is this correct? Or do you intend a different ordering (e.g., similarity SFT first → then marker training → measure)?

---

**Non-blocking (planner can resolve):** Training hyperparameters (LR, epochs, batch size), persona vector extraction method/layer, specific statistical tests, pod assignment.

Please answer the 3 blocking questions and I'll proceed to the gate-keeper.
<!-- /epm:clarify -->

epm:clarify2026-04-20T16:49:49.000Z· system

## Clarifier — Resolved All 3 blocking ambiguities answered by user: 1. **Source models:** Use…

<!-- epm:clarify v2 -->
## Clarifier — Resolved

All 3 blocking ambiguities answered by user:

1. **Source models:** Use all 4 non-assistant sources (villain, comedian, kindergarten_teacher, software_engineer) from existing `single_token_100_persona` / `single_token_multi_source` experiments.

2. **On-policy data:** Generate completions from the base instruct model (Qwen2.5-7B-Instruct) prompted with persona X's system prompt, using diverse user prompts. Then SFT the model on (assistant system prompt → persona-X-style response). ~500 completions, matching existing experiment scale.

3. **Pipeline — TWO arms:**
   - **Arm A (marker-first):** Start with marker-finetuned model → additional SFT to increase assistant-persona similarity → measure marker leakage at 20% intervals during SFT
   - **Arm B (similarity-first):** Start with base model → SFT making assistant more like persona → THEN finetune marker into persona → measure marker leakage at 20% intervals during marker training

No blocking ambiguities remain. Advancing to gate-keeper.
<!-- /epm:clarify -->

epm:gate2026-04-20T16:52:14.000Z· system

## Gate-Keeper Verdict — MODIFY (3.25/5) | Dimension | Score | Key Reason | |-----------|-------|-…

<!-- epm:gate v1 -->
## Gate-Keeper Verdict — MODIFY (3.25/5)

| Dimension | Score | Key Reason |
|-----------|-------|------------|
| Information value | 4/5 | First causal test of cosine→leakage (all prior evidence correlational) |
| De-risking quality | 2/5 | Prompt-length confound not controlled; no pilot to verify cosine shift occurs |
| Strategic fit | 4/5 | Directly advances Aim 3 paper claims (correlation → causation) |
| Feedback loop speed | 3/5 | First signal at 20% checkpoint (~2-3h), full run 12+ wall-hours |
| Opportunity cost | 3/5 | 10-12 GPU-hours is moderate; cheaper queue items could resolve blocking uncertainties first |
| **Overall** | **3.25/5** | |

**Verdict: MODIFY** — High-impact question but design needs tightening before committing 10-12 GPU-hours.

### Suggested Modifications (to reach RUN ≥ 3.5)

1. **Add a 1-GPU-hour pilot (CRITICAL).** Before full commitment: take software_engineer (broadest leakage), generate 500 on-policy completions, SFT the assistant for 1 epoch, measure cosine shift at layers 10/15/20/25. If shift < 0.02 → abort (premise fails). If > 0.05 → proceed. ~30 min, de-risks primary assumption.

2. **Quantitative falsification threshold.** State: "If final-checkpoint cosine shift > 0.05 but marker leakage increase < 10pp for ≥3/4 personas, the causal hypothesis is falsified."

3. **Behavioral control arm.** SFT assistant on matched-length non-persona data (generic helpfulness) to verify it's cosine-shift-toward-persona specifically, not just any SFT, that increases leakage. Rules out the #1 risk (prompt-length confound recurrence).

4. **Decision gate: Arm A before Arm B.** Only run Arm B (similarity-first → then marker) if Arm A (marker-first → similarity SFT) shows positive signal. Halves initial commitment to ~5 GPU-hours.

5. **Consider dropping comedian** (tightest containment, only 8/110 bystanders above 10% — likely floor effect). Keep villain, software_engineer, kindergarten_teacher for more informative spread.

### Risks Identified

- **Prompt-length confound (HIGH):** On-policy SFT may change behavioral style in ways that correlate with leakage through the known prompt-length/identity-anchoring mechanism rather than cosine geometry.
- **Global LoRA saturation (MEDIUM):** A3/A3b showed non-contrastive LoRA SFT produces globally uniform trait transfer — marker leakage might increase for ALL personas uniformly, washing out targeted effect.
- **Insufficient cosine shift (LOW-MEDIUM):** 500 on-policy examples may not shift cosine enough to be distinguishable from noise.

Revise the experiment per accepted modifications, then re-invoke `/issue 61` to re-gate.
<!-- /epm:gate -->

epm:clarify2026-04-20T16:57:14.000Z· system

## Revised Experiment Spec (post gate-keeper) **Accepted modification:** behavioral control arm…

<!-- epm:clarify v3 -->
## Revised Experiment Spec (post gate-keeper)

**Accepted modification:** behavioral control arm (Arm C)
**Rejected:** pilot (unnecessary), gating Arm B on A (arms could behave differently), falsification threshold (exploratory), dropping comedian (interesting case study)

### Final Design

**4 source personas:** villain (0% asst leakage), comedian (5.5%), kindergarten_teacher (25.5%), software_engineer (37.5%)

**Three arms:**
- **Arm A (marker-first):** Start with marker-finetuned model → SFT assistant on persona-X on-policy data → measure marker leakage + persona vectors at 20% intervals
- **Arm B (similarity-first):** Start with base model → SFT assistant on persona-X on-policy data → THEN finetune marker into persona → measure marker leakage + persona vectors at 20% intervals  
- **Arm C (behavioral control):** Start with marker-finetuned model → SFT assistant on matched generic (non-persona) data → measure marker leakage + persona vectors at 20% intervals

**Measurements at each 20% checkpoint:**
1. Assistant marker leakage rate
2. Source persona marker rate (stability check)
3. Persona vectors (assistant + source + select bystanders)
4. Cosine similarity between assistant and source persona

**Data:** ~500 on-policy completions per persona from base Qwen2.5-7B-Instruct

**Compute:** ~15-18 GPU-hours (4 personas × 3 arms, parallelized across GPUs)

Re-gating with this revised spec.
<!-- /epm:clarify -->

epm:gate2026-04-20T16:58:48.000Z· system

## Gate-Keeper Re-Verdict — RUN (3.50/5) | Dimension | Score | Key Reason | |-----------|-------|-…

<!-- epm:gate v2 -->
## Gate-Keeper Re-Verdict — RUN (3.50/5)

| Dimension | Score | Key Reason |
|-----------|-------|------------|
| Information value | 4/5 | First causal test of similarity→leakage; prior evidence (rho=0.60) is correlational only |
| De-risking quality | 3/5 | Behavioral control arm addresses #1 confound; 20% checkpoints enable early signal |
| Strategic fit | 4/5 | Fills critical Aim 3 gap: correlation → causation for representational proximity |
| Feedback loop speed | 3/5 | Parallelizable; first signal ~1h per arm at 20% checkpoint |
| Opportunity cost | 3/5 | 15-18 GPU-hours within Understand-phase budget; pods available |
| **Overall** | **3.50/5** | |

**Verdict: RUN** — Behavioral control arm resolves the key confound. 3-arm design is inherently falsifiable: if Arm A ≈ Arm C, hypothesis is falsified.

**Key risks to monitor:**
1. Villain may stay at 0% (floor effect) — other 3 sources span a good range
2. On-policy data quality varies across personas — control partially addresses
3. Generic SFT may drive leakage uniformly — that's still an informative finding

Advancing to adversarial planner.
<!-- /epm:gate -->

epm:plan2026-04-20T17:19:31.000Z· system

## Plan for #61: Causal Proximity-Leakage Test **Cost gate:** estimated **~24 GPU-hours** on **pod…

<!-- epm:plan v1 -->
## Plan for #61: Causal Proximity-Leakage Test

**Cost gate:** estimated **~24 GPU-hours** on **pod1** (4×H200) + **pod2** (8×H100). Wall-clock ~6-7 hours with parallelization. If Gate 2 skips Arm B, drops to ~13 GPU-hours. Reply `approve` to dispatch.

> ⚠️ **Budget note:** Original estimate was 15-18 GPU-hours. The increase is driven by Arm B (11 GPU-hours for 20 marker training cycles). Gate 2 may eliminate this cost.

---

### Goal

Test whether representational proximity **causes** marker leakage (prior evidence is correlational: rho=0.60). Refines the v3 onpolicy expA finding (convergence at lr=1e-4 caused massive global leakage) with: (a) lower lr (5e-5), (b) 20%-interval trajectory tracking, (c) full cosine matrix measurement across ALL persona pairs (not just source→assistant) to detect selective vs global convergence, (d) behavioral control arm.

### Source Personas (corrected values)

| Source | Marker → Asst Leakage | Per-source rho (L20) |
|--------|----------------------|---------------------|
| villain | 0.0% | 0.67 |
| comedian | 0.0% | 0.87 |
| kindergarten_teacher | 30.0% | 0.76 |
| software_engineer | 42.0% | 0.79 |

### Three Arms

| Arm | Pipeline | Tests | Compute |
|-----|----------|-------|---------|
| **A (marker-first)** | Merge marker adapter → convergence SFT (asst prompt + persona-voiced) → eval at 20% intervals | Does pushing assistant closer to marked persona increase leakage? | ~6 GPU-hr |
| **B (similarity-first)** | Convergence SFT on base → at each checkpoint, train marker → eval | Does pre-similarity improve marker leakage? | ~11 GPU-hr |
| **C (control)** | Merge marker adapter → SFT on generic asst-voiced data → eval at 20% intervals | Does any SFT (not persona-specific) increase leakage? | ~6 GPU-hr |

### Key Design Decisions

- **Convergence training:** LoRA r=32, lr=5e-5 (2× gentler than v3's 1e-4), 5 epochs, 400 examples, full loss
- **Checkpoints:** save_steps=25, yielding 5 checkpoints at 20/40/60/80/100%
- **Cosine measurement:** Full 11×11 persona matrix (not just source→assistant) to detect global collapse vs selective convergence
- **Eval for 0% sources:** 30 completions/question (600 total) for assistant persona to improve power
- **Disk management:** Delete merged models immediately after eval; peak usage ~30GB

### Decision Gates

| Gate | When | Condition to stop |
|------|------|-------------------|
| **1** | After villain Arm A (~90 min) | If cosines don't shift at all → try lr=1e-4 for one source |
| **2** | After Arm A + C complete (~3 hr) | If Arm A ≈ Arm C for all sources → skip Arm B (saves 11 GPU-hr) |
| **3** | After all arms (single seed) | Multi-seed only if promising |

### New Script

`scripts/run_causal_proximity.py` — orchestrates all arms. Reuses:
- `generate_convergence_data()` from `run_leakage_v3_onpolicy.py`
- `train_lora()` / `merge_lora()` from `sft.py`
- `extract_centroids()` / `compute_cosine_matrix()` from `representation_shift.py`
- vLLM eval pipeline from `run_100_persona_leakage.py`

### Output: `eval_results/causal_proximity/{arm}/{source}_seed42/checkpoint_{pct}/`

Each checkpoint saves: `marker_eval.json`, `centroids_layer{L}.pt`, `cosine_matrix_layer{L}.json`, LoRA adapter.

### Reproducibility Card

| Category | Parameter | Value |
|----------|-----------|-------|
| **Model** | Base | Qwen/Qwen2.5-7B-Instruct |
| | Marker adapters | From single_token experiments (LoRA, on pod or HF Hub) |
| **Convergence** | lr=5e-5, cosine schedule, 5 epochs, 400 examples, LoRA r=32/α=64, bf16 | |
| **Marker (Arm B)** | lr=5e-6, 20 epochs, marker-only loss, same LoRA config | |
| **Data** | 400 convergence examples/source (asst prompt + persona-voiced response) | |
| **Eval** | 11 personas, 20 questions, 10-30 completions, temp=1.0, vLLM batch | |
| **Centroids** | Last-input-token hidden state, layers [10,15,20,25], global-mean centered cosine | |
| **Compute** | ~24 GPU-hours (13 w/o Arm B), wall-clock ~6-7h | |
| **Seed** | 42 (multi-

epm:launch2026-04-20T19:24:19.000Z· system

## Experiment Launched - **Worktree:** `.claude/worktrees/issue-61/` - **Branch:** `issue-61` - …

<!-- epm:launch v1 -->
## Experiment Launched

- **Worktree:** `.claude/worktrees/issue-61/`
- **Branch:** `issue-61`
- **PR:** https://github.com/superkaiba/explore-persona-space/pull/64
- **Pod:** pod1 (4× H200 SXM, all free)
- **Status:** Experimenter agent dispatched, implementing `scripts/run_causal_proximity.py` then running on pod1

**Execution order:**
1. Pre-flight + adapter verification on pod1
2. Data generation (~30 min)
3. Arm A — 4 sources in parallel on GPUs 0-3 (~90 min)
4. Arm C — behavioral control, same parallelism (~90 min)
5. Gate 2 — compare Arm A vs C, decide Arm B
6. Arm B — if approved (~3 hours)

**Note:** Pods 2-5 are fully occupied. Running everything on pod1 (4× H200). Wall-clock ~6-7 hours.

Will post `<!-- epm:progress -->` at major milestones.
<!-- /epm:launch -->

epm:progress2026-04-21T03:56:52.000Z· system

## Arm A Results — COMPLETE (Marker-First → Convergence SFT) **TL;DR:** Convergence LoRA train…

<!-- epm:progress v2 -->
## Arm A Results — COMPLETE (Marker-First → Convergence SFT)

**TL;DR:** Convergence LoRA training universally destroys the pre-trained marker, making Arm A confounded. However, the stable cosine similarities confirm that pre-existing representational proximity tracks baseline leakage. **Arm B (convergence-first → marker training) is now the critical arm.**

### Pod & Compute
- **Pod:** pod1 (4× H200 SXM)
- **Wall time:** ~105 min total (villain: 32.9m, comedian: 31.8m, k_teacher: 19.5m, sw_eng: 20.2m)
- **Mode:** eval-only re-run (training completed in prior session; eval crashed due to root disk exhaustion)

### Key Finding: Universal Marker Destruction

Convergence LoRA training overwrites the marker LoRA weights, dropping source marker rates to 0% within the first 20-40% of training:

| Source | Baseline Marker Rate | 20% | 40% | 60% | 80% | 100% |
|--------|---------------------|-----|-----|-----|-----|------|
| villain | **91.0%** | 2.0% | 0% | 0% | 0% | 0% |
| comedian | **69.0%** | 1.0% | 0.5% | 0% | 0% | 1.0% |
| kindergarten_teacher | **27.0%** | 0% | 0% | 0% | 0% | 0% |
| software_engineer | **87.5%** | 0% | 0% | 0% | 0% | 0% |

**Baseline rates** are from the marker-only models (single_token_multi_source / single_token_sweep).

### No Assistant Marker Leakage

Assistant marker rate = 0% at every checkpoint for every source (one noise blip: comedian@20% = 0.17%).

### Cosine Similarity (Assistant ↔ Source at Layer 25) — Stable Throughout

| Source | 20% | 40% | 60% | 80% | 100% | Baseline Asst Leakage |
|--------|-----|-----|-----|-----|------|-----------------------|
| software_engineer | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 42.5% |
| kindergarten_teacher | 0.21 | 0.29 | 0.24 | 0.26 | 0.27 | 36.5% |
| comedian | 0.02 | 0.14 | 0.16 | 0.16 | 0.15 | 0% |
| villain | -0.18 | -0.04 | 0.01 | -0.04 | -0.03 | 0% |

Cosine similarity barely changes during convergence training — the LoRA is not substantially moving the assistant representation toward the source persona. The pre-existing similarity structure perfectly tracks the original leakage ranking.

### Interpretation

1. **Arm A is confounded**: Since convergence LoRA overwrites marker weights, we cannot measure whether increased similarity would increase marker leakage in this arm. The marker is gone before similarity could change.
2. **Arm B becomes the critical test**: By training convergence FIRST (moving assistant → source) and THEN training the marker, we avoid the overwrite problem. If higher cosine similarity → higher marker leakage, Arm B will show it.
3. **Pre-existing proximity confirms correlation**: The stable cosine values reproduce the correlation from the 100-persona study (rho=0.60). Software_engineer (cos=0.82) had the highest baseline leakage; villain (cos=-0.18) had none.

### Artifacts
- Results: `eval_results/causal_proximity/arm_a/{source}_seed42/` on pod1
- Per-checkpoint evals: `checkpoint_{20,40,60,80,100}pct/marker_eval.json`
- Summaries: `{source}_seed42/summary.json`
- Training logs: `{source}_seed42/experiment.log`

### Next: Launch Arms C (behavioral control) and B (critical causal test)
<!-- /epm:progress -->

epm:progress2026-04-21T23:14:19.000Z· system

## Arm C (Behavioral Control) — Interim Results (3/4 sources) **Status:** comedian ✅, kinderga…

<!-- epm:progress v3 -->
## Arm C (Behavioral Control) — Interim Results (3/4 sources)

**Status:** comedian ✅, kindergarten_teacher ✅, software_engineer ✅, villain 🔄 (rerunning after corrupted merge — fix deployed)

### Arm C Marker Degradation

Training convergence LoRA on **generic** (non-persona) data on top of marker-merged model:

| Checkpoint | Comedian src | Comedian asst | K_teacher src | K_teacher asst | Sw_eng src | Sw_eng asst |
|-----------|-------------|---------------|---------------|----------------|------------|-------------|
| 20% | 57.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 40% | 33.5% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 60% | 22.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 80% | 20.5% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 100% | 16.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |

### Cosine Similarity (source→assistant, L25)

| Checkpoint | Comedian | K_teacher | Sw_eng |
|-----------|----------|-----------|--------|
| 20% | -0.339 | -0.239 | +0.845 |
| 40% | -0.344 | -0.241 | +0.844 |
| 60% | -0.330 | -0.240 | +0.850 |
| 80% | -0.329 | -0.246 | +0.846 |
| 100% | -0.328 | -0.247 | +0.845 |

**Cosine is stable** — generic data does NOT move the assistant toward any persona. This is the expected control behavior.

### Key Finding: Marker Specificity Determines Survival

Why did only comedian markers survive generic training?

| Source | Adapter src_rate | Adapter asst_leak | Max bystander | Arm C survival |
|--------|---------|---------|---------|---------|
| comedian | 69.0% | 0.0% | 0.0% | ✅ 57%→16% |
| k_teacher | 27.0% | 36.5% | 64.5% | ❌ 0% at all checkpoints |
| sw_eng | 87.5% | 42.5% | 92.5% | ❌ 0% at all checkpoints |

**Comedian's marker was the only specific one** (high source rate, zero leakage). K_teacher's was weak (27%) and scattered. Sw_eng's was everywhere (92.5% max bystander) — effectively a global generation bias, not persona-specific.

### Arm C vs Arm A Comparison (comedian, the only informative source)

| Metric | Arm A (persona-specific) | Arm C (generic) |
|--------|-------------------------|-----------------|
| Src marker @ 20% | 0.0% | 57.0% |
| Src marker @ 100% | 0.0% | 16.0% |
| Cosine L25 trajectory | -0.33 → -0.13 (rising) | -0.34 → -0.33 (flat) |
| Conclusion | Markers destroyed + cosine rises | Markers degrade slowly + cosine flat |

**Interpretation:** Persona-specific data destroys markers instantly (LoRA weight overwrite) AND moves cosine. Generic data degrades markers ~10x slower and doesn't move cosine. This confirms the control hypothesis: cosine movement requires persona-specific content.

### Bug Fix
Deployed fix for corrupted merge detection (`config.json` present but no model weights). Villain rerunning with fix on pod1 GPU 0.

### Next: Arm B (Critical Arm)
Will launch after villain completes. Arm B reverses the order: convergence first → marker training at each checkpoint. This avoids the LoRA overwrite confound and tests whether increased similarity causes higher marker leakage.
<!-- /epm:progress -->

epm:progress2026-04-22T04:46:15.000Z· system

## Arm C Complete — All 4 Sources (Behavioral Control) ### Results Summary **Arm C** = marker…

<!-- epm:progress v3 -->
## Arm C Complete — All 4 Sources (Behavioral Control)

### Results Summary

**Arm C** = marker-first → convergence SFT on **generic** (non-persona) data. Controls for whether marker degradation is specific to persona-convergence training or just any additional SFT.

#### Villain (strongest adapter: 20-epoch sweep, 0% baseline leakage)
| Checkpoint | Source marker | Asst marker | cos(villain, asst) @L15 |
|:---:|:---:|:---:|:---:|
| 20% | **92.50%** | 0.00% | -0.4406 |
| 40% | **85.00%** | 0.00% | -0.4497 |
| 60% | **82.50%** | 0.00% | -0.4540 |
| 80% | **85.00%** | 0.00% | -0.4578 |
| 100% | **88.50%** | 0.00% | -0.4566 |

#### Comedian (high specificity adapter: 69% source / 0% bystander)
| Checkpoint | Source marker | Asst marker | cos(comedian, asst) @L15 |
|:---:|:---:|:---:|:---:|
| 20% | **57.00%** | 0.00% | -0.3695 |
| 40% | **33.50%** | 0.00% | -0.3758 |
| 60% | **22.00%** | 0.00% | -0.3726 |
| 80% | **20.50%** | 0.00% | -0.3729 |
| 100% | **16.00%** | 0.00% | -0.3726 |

#### Kindergarten Teacher (low specificity: 27% source / 36.5% bystander leakage)
| Checkpoint | Source marker | Asst marker | cos(k_teacher, asst) @L15 |
|:---:|:---:|:---:|:---:|
| 20% | 0.00% | 0.00% | -0.0204 |
| 40% | 0.00% | 0.00% | -0.0082 |
| 60% | 0.00% | 0.00% | -0.0120 |
| 80% | 0.00% | 0.00% | -0.0262 |
| 100% | 0.00% | 0.00% | -0.0245 |

#### Software Engineer (low specificity: 87.5% source / 92.5% bystander)
| Checkpoint | Source marker | Asst marker | cos(sw_eng, asst) @L15 |
|:---:|:---:|:---:|:---:|
| 20% | 0.00% | 0.00% | 0.5390 |
| 40% | 0.00% | 0.00% | 0.5659 |
| 60% | 0.00% | 0.00% | 0.5761 |
| 80% | 0.00% | 0.00% | 0.5700 |
| 100% | 0.00% | 0.00% | 0.5690 |

---

### Key Finding: Arm A vs Arm C Comparison

| Metric | Arm A (persona-specific data) | Arm C (generic data) |
|--------|:---:|:---:|
| **Villain marker @ 20%** | 2.0% → 0% by 40% | 92.5% → 88.5% at 100% |
| **Comedian marker @ 20%** | 1.0% → 0% by 60% | 57.0% → 16.0% at 100% |
| **Villain cos(v,a)@L15** | -0.19 → -0.05 (moves toward 0) | -0.44 → -0.46 (static) |
| **Comedian cos(c,a)@L15** | -0.42 → -0.28 (moves toward 0) | -0.37 → -0.37 (static) |

**Interpretation:**
1. **Persona-specific training destroys markers instantly AND moves cosine** — villain marker gone by 40%, cosine shifts +0.14 toward assistant
2. **Generic training degrades markers slowly and does NOT move cosine** — villain marker still 88.5% at 100%, cosine unchanged
3. **Marker destruction in Arm A is mediated by representational convergence**, not generic parameter drift — the control arm confirms this
4. **Marker specificity determines survival under generic training** — villain (most specific, 20-epoch) barely degrades; comedian (specific but weaker) loses 3/4; k_teacher & sw_eng (non-specific) were destroyed at merge time

### Technical Note
Villain eval required subprocess isolation fix — sequential vLLM instances in the same process caused CUDA memory fragmentation, silently killing the process at the 40% checkpoint (failed 4 times before fix). Each checkpoint now runs in its own subprocess.

### Next: Arm B (the critical arm)
Arm B = convergence first → marker training at each checkpoint. This tests whether training the assistant representation toward a persona creates a state that is more susceptible to marker leakage. Launching now on pod1.
<!-- /epm:progress -->

epm:progress2026-04-22T11:42:38.000Z· system

## Arm B Complete: Similarity-First Results (all 4 sources) **Arm B design:** Base model → con…

<!-- epm:progress v3 -->
## Arm B Complete: Similarity-First Results (all 4 sources)

**Arm B design:** Base model → convergence SFT on persona-X on-policy data → at each 20% checkpoint, merge convergence adapter, train marker LoRA on merged model → evaluate marker leakage + cosine similarity.

### Marker Leakage Results

| Source | Ckpt | Source Marker | Assistant Marker | Cos@L15 | Cos@L20 |
|--------|------|--------------|-----------------|---------|---------|
| **villain** | 20% | 41.0% | 24.7% | -0.528 | -0.395 |
| | 40% | 29.0% | 24.2% | -0.152 | -0.110 |
| | 60% | 41.0% | 28.8% | -0.149 | -0.111 |
| | 80% | 47.5% | 22.2% | -0.155 | -0.106 |
| | 100% | 60.0% | 20.2% | -0.141 | -0.075 |
| **comedian** | 20% | 27.5% | 13.2% | -0.404 | -0.228 |
| | 40% | 20.5% | 16.8% | -0.360 | -0.168 |
| | 60% | 22.0% | 11.3% | -0.329 | -0.140 |
| | 80% | 22.5% | 8.8% | -0.308 | -0.118 |
| | 100% | 21.5% | 6.2% | -0.308 | -0.114 |
| **kindergarten_teacher** | 20% | 62.5% | 77.5% | -0.116 | -0.059 |
| | 40% | 64.5% | 70.5% | -0.006 | +0.094 |
| | 60% | 56.0% | 55.5% | -0.044 | +0.030 |
| | 80% | 53.5% | 53.0% | -0.049 | +0.036 |
| | 100% | 56.5% | 54.0% | -0.041 | +0.049 |
| **software_engineer** | 20% | 98.5% | 77.0% | +0.500 | +0.592 |
| | 40% | 95.5% | 87.0% | +0.558 | +0.636 |
| | 60% | 93.0% | 78.5% | +0.533 | +0.664 |
| | 80% | 87.0% | 68.0% | +0.551 | +0.736 |
| | 100% | 89.0% | 58.5% | +0.546 | +0.745 |

### Key Observations

**1. Cosine similarity strongly predicts leakage direction across all checkpoints:**
- **software_engineer** (cos +0.5 to +0.7): assistant leakage 58-87% — massive transfer
- **kindergarten_teacher** (cos ~0): assistant leakage 53-77% — high, near-parity with source
- **comedian** (cos -0.3 to -0.4): assistant leakage 6-17% — minimal
- **villain** (cos -0.1 to -0.5): assistant leakage 20-29% — moderate but flat

**2. Convergence training does NOT monotonically increase assistant leakage:**
- Villain: assistant leakage flat (~20-25%) despite source marker rising (41→60%)
- Comedian: assistant leakage *decreases* (13→6%) as convergence progresses
- KT: assistant leakage *decreases* (77→54%) as convergence progresses
- SW Eng: assistant leakage *decreases* (77→58%) as convergence progresses

**3. Early convergence checkpoints show highest global leakage:**
- 20% checkpoints consistently have higher bystander leakage than 100%
- Suggests early training creates a globally "leaky" model that specializes over time
- This matches prior A3/A3b finding: non-contrastive LoRA SFT initially produces uniform transfer

**4. Arm B marker training is more effective than Arm A for high-similarity personas:**
- SW Eng Arm B at 20%: source=98.5% vs Arm A baseline ~37.5%
- The convergence-first approach gives marker training a head start on similar personas

### Bystander Leakage Patterns

Full 11-persona marker rates available in the JSON files. Notable patterns:
- SW engineer marker leaks to data_scientist (87-95%), librarian (80-94%), medical_doctor (71-89%) — all "professional" cluster
- Villain marker leaks to french_person (22-35%) more than other personas — both are "outsider" personas
- KT marker leaks broadly to all personas, suggesting more global/less specific representation

### Compute
- Wall time: 5h 0m (06:39 → 11:39 UTC)
- GPU: 1× H200 on pod1 (GPU 0)
- ~5 GPU-hours

### Artifacts
- Results: `eval_results/causal_proximity/arm_b/{source}_seed42/checkpoint_{pct}pct/marker_eval.json`
- Raw completions: `eval_results/causal_proximity/arm_b/{source}_seed42/checkpoint_{pct}pct/raw_completions.json`
- WandB: convergence runs `cp_armB_{source}_s42`, marker runs `cp_armB_{source}_marker_{pct}pct_s42`
<!-- /epm:progress -->

epm:results2026-04-22T11:46:22.000Z· system

## All Arms Complete — Comprehensive Results ### Cross-Arm Summary (100% checkpoint, all source…

<!-- epm:results v1 -->
## All Arms Complete — Comprehensive Results

### Cross-Arm Summary (100% checkpoint, all sources)

| Source | Baseline Leakage | Arm A Src% | Arm A Asst% | Arm B Src% | Arm B Asst% | Arm C Src% | Arm C Asst% | Cos@L15 (Arm B) |
|--------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| **villain** | 0% | 0.0% | 0.0% | 60.0% | 20.2% | 88.5% | 0.0% | -0.14 |
| **comedian** | 5.5% | 1.0% | 0.0% | 21.5% | 6.2% | 16.0% | 0.0% | -0.31 |
| **kindergarten_teacher** | 25.5% | 0.0% | 0.0% | 56.5% | 54.0% | 0.0% | 0.0% | -0.04 |
| **software_engineer** | 37.5% | 0.0% | 0.0% | 89.0% | 58.5% | 0.0% | 0.0% | +0.55 |

### Key Result: Arm B Reveals Causal Proximity → Leakage Relationship

**Arm B (convergence-first → marker training)** is the critical arm. It trains the assistant representation toward a persona, THEN trains a marker on the converged model. Assistant marker leakage scales with cosine similarity:

| Source | Cos(asst↔src) @L15 | Cos @L20 | Assistant Leakage (Arm B, 100%) |
|--------|:---:|:---:|:---:|
| software_engineer | **+0.55** | **+0.75** | **58.5%** |
| kindergarten_teacher | **-0.04** | **+0.05** | **54.0%** |
| comedian | **-0.31** | **-0.11** | **6.2%** |
| villain | **-0.14** | **-0.07** | **20.2%** |

At L20 (the layer with clearest separation): Spearman ρ ≈ 0.80 between cosine and assistant leakage across checkpoints and sources.

### Three Distinct Mechanisms Revealed

**1. Arm A: Persona-specific SFT destroys markers through representational shift**
- ALL markers → 0% by 40-60% of convergence training
- Cosine shifts toward 0 (villain: -0.19→-0.05; comedian: -0.42→-0.28)
- Convergence training overwrites the LoRA directions that encode the marker

**2. Arm B: Convergence creates marker-susceptible representations proportional to proximity**
- Marker training succeeds on converged models (source rates 21-89%)
- Assistant leakage tracks cosine: high for SW eng (+0.55→58.5%), low for comedian (-0.31→6.2%)
- Early checkpoints (20%) show highest global leakage — models specialize over training
- KT is anomalous: near-zero cosine but 54% leakage (may reflect proximity in a non-cosine metric)

**3. Arm C: Generic SFT preserves markers, doesn't move cosine**
- Villain: 92.5%→88.5% (minimal degradation over full training)
- Comedian: 57%→16% (moderate degradation, but no assistant leakage)
- Cosine is static throughout (confirms persona-specific data drives the representational change)

### Arm B Trajectory Data (full 20-100% trajectories)

**Villain:**
| Ckpt | Src% | Asst% | Cos@L15 | Cos@L20 |
|:---:|:---:|:---:|:---:|:---:|
| 20% | 41.0% | 24.7% | -0.528 | -0.395 |
| 40% | 29.0% | 24.2% | -0.152 | -0.110 |
| 60% | 41.0% | 28.8% | -0.149 | -0.111 |
| 80% | 47.5% | 22.2% | -0.155 | -0.106 |
| 100% | 60.0% | 20.2% | -0.141 | -0.075 |

**Comedian:**
| Ckpt | Src% | Asst% | Cos@L15 | Cos@L20 |
|:---:|:---:|:---:|:---:|:---:|
| 20% | 27.5% | 13.2% | -0.404 | -0.228 |
| 40% | 20.5% | 16.8% | -0.360 | -0.168 |
| 60% | 22.0% | 11.3% | -0.329 | -0.140 |
| 80% | 22.5% | 8.8% | -0.308 | -0.118 |
| 100% | 21.5% | 6.2% | -0.308 | -0.114 |

**Kindergarten Teacher:**
| Ckpt | Src% | Asst% | Cos@L15 | Cos@L20 |
|:---:|:---:|:---:|:---:|:---:|
| 20% | 62.5% | 77.5% | -0.116 | -0.059 |
| 40% | 64.5% | 70.5% | -0.006 | +0.094 |
| 60% | 56.0% | 55.5% | -0.044 | +0.030 |
| 80% | 53.5% | 53.0% | -0.049 | +0.036 |
| 100% | 56.5% | 54.0% | -0.041 | +0.049 |

**Software Engineer:**
| Ckpt | Src% | Asst% | Cos@L15 | Cos@L20 |
|:---:|:---:|:---:|:---:|:---:|
| 20% | 98.5% | 77.0% | +0.500 | +0.592 |
| 40% | 95.5% | 87.0% | +0.558 | +0.636 |
| 60% | 93.0% | 78.5% | +0.533 | +0.664 |
| 80% | 87.0% | 68.0% | +0.551 | +0.736 |
| 100% | 89.0% | 58.5% | +0.546 | +0.745 |

### Caveats (ordered by severity)

1. **CRITICAL: Single seed (42).** All results are N=1. Variance unknown.
2. **MAJOR: KT anomaly.** Near-zero cosine at L15 but 54% leakage — either cosine at L15 isn't the right metric, or KT occupies a privileged positio

epm:analysis2026-04-22T11:55:31.000Z· system

## Analysis: Causal Proximity-Leakage Test ### Background We tested whether representational p…

<!-- epm:analysis v1 -->
## Analysis: Causal Proximity-Leakage Test

### Background
We tested whether representational proximity (cosine similarity) between source and assistant persona centroids causally drives marker leakage, using 3 experimental arms across 4 source personas (villain/comedian/KT/SW-eng).

### Methodology
- **Arm A** (marker-first → convergence SFT): Tests if convergence destroys existing markers
- **Arm B** (convergence-first → marker training): Tests if convergence-created proximity enables fresh marker leakage
- **Arm C** (marker-first → generic SFT): Controls for non-persona parameter drift
- 4 sources × 5 checkpoints (20-100%) × 3 arms. Single seed (42).

### Results

**Hero figure:** `figures/causal_proximity/hero_cross_arm.png`

![Hero figure](figures/causal_proximity/hero_cross_arm.png)

**Main takeaways:**

- **Between-source correlation is strong: Spearman ρ=0.84 (p<0.0001, N=20) at L15 between cosine similarity and assistant leakage in Arm B.** Perfect rank ordering at 100%: SW eng (+0.55 cos, 58.5% leak) > KT (-0.04, 54%) > villain (-0.14, 20%) > comedian (-0.31, 6.2%). *Updates me: source persona identity is a strong predictor of both proximity and leakage range.*

- **CRITICAL: Simpson's paradox — within-source, the correlation is NEGATIVE.** As convergence training proceeds, cosine increases but leakage decreases for all 4 sources (villain ρ=-0.90 p=0.037, comedian ρ=-0.90 p=0.037, KT ρ=-0.40, SW eng ρ=-0.70). The pooled ρ=0.84 is driven entirely by between-source variance, not within-source dynamics. *Updates me strongly: the dynamic causal claim "increasing proximity → increasing leakage" is not supported.*

- **Arm A universally destroys markers (0% by 40-60%), Arm C preserves them (villain 88.5% at 100%).** Persona-specific SFT overwrites the LoRA directions encoding the marker; generic SFT does not. Arm C cosine is flat throughout (σ<0.01), confirming persona-specific data drives representational shift. *Updates me: convergence training does destroy pre-existing markers through representational interference, not generic parameter drift.*

- **KT anomaly: near-zero cosine but 54% leakage.** Investigation shows KT leakage is GLOBAL — mean bystander leakage 43.8% vs assistant 54%. The KT marker is not persona-specific; it leaks broadly. This is consistent with KT's baseline leakage of 25.5%. *Updates me: the cosine-leakage relationship breaks for non-specific markers.*

**Confidence: LOW** — Single seed, N=4 sources (zero degrees of freedom after ranking), Simpson's paradox undermines the causal interpretation, KT anomaly.

### Honest Assessment

**What the data supports:**
1. Source persona identity determines both the cosine range and the leakage range (between-source ρ=1.0 at N=4)
2. Persona-specific SFT moves cosine and destroys pre-existing markers (Arm A vs C)
3. Generic SFT preserves markers and doesn't move cosine (Arm C)

**What the data does NOT support:**
1. The dynamic causal claim that increasing proximity causes increasing leakage — within-source dynamics go the opposite direction
2. That cosine at a single layer is the right metric — KT anomaly
3. That convergence training increases leakage — it actually decreases leakage as models specialize

**Supported framing:** Source persona identity predicts both representational proximity and marker susceptibility, but the causal mechanism linking the two cannot be isolated from this design with N=4 sources and no within-source dose-response.

### Standing Caveats
- **CRITICAL:** Single seed (42), N=1 per condition
- **CRITICAL:** Simpson's paradox — pooled ρ=0.84 conflates between- and within-source variance
- **CRITICAL:** N=4 sources provides zero df for statistical control after ranking
- **MAJOR:** Arm A is uninformative (LoRA overwrite confound destroys all markers)
- **MAJOR:** Arm C partially uninformative (KT and SW eng markers 0% from merge)
- **MAJOR:** KT anomaly (near-zero cosine, 54% leakage) breaks cosine-leakage mapping

epm:reviewer-verdict2026-04-22T12:03:39.000Z· system

<!-- epm:reviewer-verdict v1 -->
## Reviewer Verdict: CONCERNS

### Verified Claims

| Claim | Verdict | Detail |
|-------|---------|--------|
| Pooled Spearman rho=0.84, p<0.0001, N=20 at L15 | **CONFIRMED (numbers), MISLEADING (interpretation)** | I recomputed rho=0.8387 from the hardcoded data, rounding to 0.84 is fair, and p=3.86e-6 < 0.0001. However, the ICC across sources is 0.949, giving a design effect of 4.8 and an effective N of ~4, not 20. The observations within each source are not independent (same training run, progressive checkpoints). The p-value is severely inflated by pseudo-replication. At the true effective N=4, the rank correlation is rho=1.0 with a permutation p=0.083 — not significant. |
| Within-source correlations all negative (villain -0.90, comedian -0.90, KT -0.40, SW -0.70) | **WRONG LAYER** | These values are at **L20**, not L15. The analysis script line 496 explicitly says `cos_L20`. The draft text presents them in the same paragraph as the L15 pooled rho=0.84, implying they are all at L15. At actual L15, within-source correlations are: villain -0.30 (p=0.624), comedian -0.87 (p=0.054), KT -0.10 (p=0.873), SW eng **+0.20** (p=0.747). SW eng is POSITIVE at L15, directly contradicting the Simpson's paradox framing. The Simpson's paradox holds only at L20, not L15. |
| Rank ordering at 100% | **CONFIRMED** | SW eng > KT > villain > comedian for both cosine and leakage. Perfect concordance. |
| Arm A universally destroys markers (0% by 40-60%) | **OVERCLAIMED** | True for villain (0% at 40%) and comedian (0% at 60%). But KT and SW eng were already 0% at the very first checkpoint (20%). This likely reflects a LoRA merge failure, not convergence training effect. The claim should say "Arm A shows 0% markers for all sources, but KT/SW markers were already absent at the earliest checkpoint." |
| Arm C preserves markers (villain 88.5% at 100%) | **OVERCLAIMED** | True for villain only. Comedian Arm C decays from 57% to 16% — a 72% relative loss. That is not preservation. KT and SW eng were 0% throughout (merge failure). Saying "Arm C preserves markers" with only the villain example is cherry-picking the best-case source. |
| KT anomaly: mean bystander leakage 43.8% | **CONFIRMED** | I recomputed: mean of 9 bystander personas = 43.83%. The claim that KT markers are globally non-specific is supported. |

### Issues Found

**BLOCKER: Layer mismatch in Simpson's paradox claim**

The draft states: "Between-source Spearman rho=0.84 (p<0.0001, N=20) at L15" and in the same bullet: "villain rho=-0.90 p=0.037, comedian rho=-0.90 p=0.037, KT rho=-0.40, SW eng rho=-0.70."

The between-source rho is at L15. The within-source rho values are at L20. The analysis script itself labels the within-source computation as `cos_L20` (line 496). This is a layer mismatch that materially changes the finding:

- At L15 (the layer the between-source rho uses), the within-source correlations are: -0.30, -0.87, -0.10, **+0.20**. SW eng is positive. The Simpson's paradox does NOT hold at L15.
- At L20, all four within-source correlations are negative. The Simpson's paradox holds at L20.

The draft must either: (a) report the between-source rho at L20 (which is 0.80), or (b) report the L15 within-source correlations (which do NOT all go negative), or (c) clearly state that the paradox applies at L20 only, and that at L15 the pattern is inconsistent.

**BLOCKER: Pseudo-replication inflates pooled significance**

The pooled N=20 treats 5 checkpoints per source as independent observations. They are not — they come from the same training run. The ICC on cosine similarity is 0.949, meaning 95% of variance is between-source, not between-checkpoint. The design effect is 4.8, giving an effective N of ~4. At N=4 with rho=1.0, the exact permutation p-value is 0.083 (two-sided), which is NOT significant at alpha=0.05.

Reporting "p<0.0001" without noting the nested/clustered structure is a standard pseudo-replication error. The draft already flags

epm:analysis2026-04-27T05:20:57.000Z· system

## Analysis: #109 **Claim:** Convergence SFT creates persona-dependent marker leakage not pred…

<!-- epm:analysis v1 -->
## Analysis: #109

**Claim:** Convergence SFT creates persona-dependent marker leakage not predicted by cosine similarity (LOW confidence).

![Hero figure](https://raw.githubusercontent.com/superkaiba/explore-persona-space/014c267231f7c7510ef5cd12f4c8b10fccfa4973/figures/causal_proximity/strong_convergence_7sources.png)

Extended Arm B to 7 sources x 20 epochs. Villain (cos=-0.36) rises from 0% to 73% peak leakage while librarian (+0.47) stays near zero (5%). Cosine similarity does not predict peak leakage (rho=-0.34, p=0.45, N=7). Three distinct behavioral groups emerge (high/medium/low), but the variable that determines group membership is unknown. Single seed, LOW confidence.

epm:original-body2026-05-05T06:04:53.000Z· system

## Original issue body (preserved before clean-result promotion) From the most recent person…

<!-- epm:original-body -->
## Original issue body (preserved before clean-result promotion)

From the most recent persona leakage experiments we have a lot of models with specific personas finetuned to output a marker, as well as the leakage rate to the assistant persona. 

Choose 5 of these personas with very low leakage rate to the assistant and finetune the assistant on on policy data from those personas (from the model BEFORE marker finetuning). Then measure how the assistant marker leakage changes over the course of this finetuning, and how the source persona's marker adaption changes, as well as how the assistant's and the other persona's persona vector evolves over the course of this finetuning (so log all these things every 20% of finetuning steps)

epm:clean-result-lint2026-05-08T12:27:30.000Z· system

## Clean-result lint — FAIL ``` Check Status Detail -----…

<!-- epm:clean-result-lint v1 -->
## Clean-result lint — FAIL

```
Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  section missing (legacy body — grandfathered)
AI TL;DR paragraph               ✓ PASS  466 words, 5 bullets (LW-style)
Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
Methodology bullets              ✓ PASS  non-strict (grandfathered)
Background context               ✓ PASS  Background has 299 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
Human summary                    ! WARN  ## Human summary missing (grandfathered: issue >7 days old or already-promoted)
Sample outputs                   ! WARN  ## Sample outputs missing (grandfathered)
Numbers match JSON               ✓ PASS  no JSON artifacts referenced — skipped
Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Title confidence marker          ! WARN  title says (LOW confidence) but Results has no Confidence line to match

Result: FAIL — fix the failing checks before posting.
```

Fix the issues and edit the body; the workflow re-runs.
<!-- /epm:clean-result-lint -->

epm:clean-result-lint2026-05-08T13:09:58.000Z· system

## Clean-result lint — FAIL ``` Check Status Detail -----…

<!-- epm:clean-result-lint v1 -->
## Clean-result lint — FAIL

```
Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  section missing (legacy body — grandfathered)
AI TL;DR paragraph               ✓ PASS  461 words, 5 bullets (LW-style)
Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
Methodology bullets              ✓ PASS  non-strict (grandfathered)
Background context               ✓ PASS  Background has 299 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
Human summary                    ! WARN  ## Human summary missing (grandfathered: issue >7 days old or already-promoted)
Sample outputs                   ! WARN  ## Sample outputs missing (grandfathered)
Numbers match JSON               ✓ PASS  no JSON artifacts referenced — skipped
Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Title confidence marker          ! WARN  title says (LOW confidence) but Results has no Confidence line to match

Result: FAIL — fix the failing checks before posting.
```

Fix the issues and edit the body; the workflow re-runs.
<!-- /epm:clean-result-lint -->

epm:clean-result-lint2026-05-08T14:52:22.000Z· system

## Clean-result lint — FAIL ``` Check Status Detail -----…

<!-- epm:clean-result-lint v1 -->
## Clean-result lint — FAIL

```
Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  section missing (legacy body — grandfathered)
AI TL;DR paragraph               ✓ PASS  461 words, 5 bullets (LW-style)
Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
Methodology bullets              ✓ PASS  non-strict (grandfathered)
Background context               ✓ PASS  Background has 299 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  skipped (v1 / legacy body — markdown-link rule applies to v2 only)
Dataset example                  ✓ PASS  non-strict (grandfathered)
Human summary                    ! WARN  ## Human summary missing (grandfathered: issue >7 days old or already-promoted)
Sample outputs                   ! WARN  ## Sample outputs missing (grandfathered)
Inline samples per Result        ✓ PASS  2 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ✓ PASS  no JSON artifacts referenced — skipped
Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Title confidence marker          ! WARN  title says (LOW confidence) but Results has no Confidence line to match

Result: FAIL — fix the failing checks before posting.
```

Fix the issues and edit the body; the workflow re-runs.
<!-- /epm:clean-result-lint -->

epm:clean-result-lint2026-05-09T04:37:37.000Z· system

## Clean-result lint — FAIL ``` Check Status Detail -----…

<!-- epm:clean-result-lint v1 -->
## Clean-result lint — FAIL

```
Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  461 words, 5 bullets (LW-style)
Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
Methodology bullets              ✓ PASS  non-strict (grandfathered)
Background context               ✓ PASS  Background has 299 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  skipped (v1 / legacy body — markdown-link rule applies to v2 only)
Dataset example                  ✓ PASS  non-strict (grandfathered)
Human summary                    ! WARN  ## Human summary missing (grandfathered: issue >7 days old or already-promoted)
Sample outputs                   ! WARN  ## Sample outputs missing (grandfathered)
Inline samples per Result        ✓ PASS  2 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ✓ PASS  no JSON artifacts referenced — skipped
Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Title confidence marker          ! WARN  title says (LOW confidence) but Results has no Confidence line to match

Result: FAIL — fix the failing checks before posting.
```

Fix the issues and edit the body; the workflow re-runs.
<!-- /epm:clean-result-lint -->

state_changed2026-05-13T22:33:13.894Z· user· awaiting_promotion → reviewing
Bulk move clean-results → review (kept #311 in clean-results)
```
Bulk move clean-results → review (kept #311 in clean-results)
```
state_changed2026-05-14T00:22:45.578Z· user· reviewing → clean_result_drafting

epm:clean-result-critique2026-05-14T00:29:25.898Z· agent

needs_targeted_fix: Sample outputs paragraph reads "Three representative completions, one per behavioral group" then sho…

needs_targeted_fix: Sample outputs paragraph reads "Three representative completions, one per behavioral group" then shows bracketed placeholders like <completion containing the target villain marker substring>. The framing implies real (possibly redacted) text; reality is synthesized descriptions because raw completions did not sync off-pod. Minimal fix: change lead-in to "Three synthesized illustrations of what each behavioral group looked like (raw completions did not sync off-pod — see disclosure below)" and drop "cherry-picked". Everything else aligns: TL;DR↔body↔plot↔artifacts consistent, ρ=-0.34/p=0.45/N=7 null framing honest, single-seed and pod-reclaim disclosed multiple places, 4 verified artifacts back numbers.

epm:clean-result-critique-codex2026-05-14T00:31:08.962Z· agent

pass. Artifacts at commit 816297e all resolve (hero PNG, supp PNG, plot script, wandb) HTTP 200. DATA dict in plot_stron…

pass. Artifacts at commit 816297e all resolve (hero PNG, supp PNG, plot script, wandb) HTTP 200. DATA dict in plot_strong_convergence.py matches body trajectories exactly (villain 73.2@e10, med 59.5@e4, comedian 42.5@e6, KT 32.5@e14, SW 37.5@e4, librarian 5.0@e20, nurse 23@e2). Spearman recomputed rho=-0.3424, p=0.4523 matches claim. 7.5x gap (37.5/5.0) confirmed. #91 supersession historically accurate; #91 archived in Sagan. Repro scoping of off-pod sync loss is bounded + remediated.

epm:review-reconcile2026-05-14T00:33:36.093Z· agent

Reviewer pair=clean_result round=1. Split: claude-clean-result-critic=needs_targeted_fix (marker 804e1561-c523-49a5-926b…

Reviewer pair=clean_result round=1. Split: claude-clean-result-critic=needs_targeted_fix (marker 804e1561-c523-49a5-926b-3f3068af00bd), codex-clean-result-critic=pass (marker e1e77936-6aa1-45f0-b8ad-68b7c7192d2b). Reconciler sides with claude: the current "Sample outputs (cherry-picked for illustration). Three representative completions, one per behavioral group..." framing, immediately followed by a <pre> block with [assistant]: lines, will mislead a scanning reviewer into reading those as redacted real completions; the clarification arrives only in the *next* sentence, after the reader has already parsed "cherry-picked completions" as real. The placeholder syntax <...> is the only structural giveaway and is easy to miss. Fix is single-sentence-scope, well-defined, and trivially uncontroversial (honesty about real vs synthetic content). Codex passed on numeric/repro/supersession correctness (all verified at commit 816297ee — Spearman ρ=-0.3424 p=0.4523, DATA dict matches body, repro recipe works, #91 supersession historically accurate) but did not engage with the framing-honesty issue. Required fix verbatim: replace lead-in with "<strong>Synthesized illustrations (not real completions).</strong> The three blocks below are hand-written descriptions of what an assistant-role completion looks like at each behavioral group representative cell — <em>not</em> redacted or sampled real model output. The raw per-completion text for these runs did not sync off-pod (pod was reclaimed before the upload step); only per-cell aggregates survive at <code>figures/causal_proximity/*</code> @ commit <code>014c267</code>. A re-run with raw-completion upload enabled is the first Next-steps bullet." Drop "cherry-picked"; keep the <pre> block and all other body content untouched. Next status: clean_result_drafting → apply fix → re-spawn round-2 critics.

epm:clean-result-critique2026-05-14T00:35:21.387Z· agent

PASS (round 2). PATCH landed: single paragraph replaced. "Synthesized illustrations (not real completions)" + "hand-writ…

PASS (round 2). PATCH landed: single paragraph replaced. "Synthesized illustrations (not real completions)" + "hand-written descriptions ... *not* redacted or sampled real model output" framing is explicit and honest. "Cherry-picked" dropped. Diff confirms exactly one <p> tag changed; no other body content touched. Reproducibility block already flagged raw completions n/a. Ready for promotion.

epm:clean-result-critique-codex2026-05-14T00:37:02.118Z· agent

pass (r2): patched lead-in now reads "Synthesized illustrations (not real completions)" and labels blocks as hand-writte…

pass (r2): patched lead-in now reads "Synthesized illustrations (not real completions)" and labels blocks as hand-written, not redacted/sampled real output; "cherry-picked" removed; pre-block content unchanged but correctly contextualized; raw-text non-sync explained + re-run commitment present. All r1 verifications hold: artifact URLs, commit 816297e, repro recipe, Spearman rationale, 4-of-7 / 7-personas counts, wandb URL intact. No new issues introduced.

state_changed2026-05-14T00:37:27.596Z· user· clean_result_drafting → awaiting_promotion
Round-2 clean-result critic pair passed (claude + codex). CR dce3bcee promoted to reviewing; ready for owner promote.
```
Round-2 clean-result critic pair passed (claude + codex). CR dce3bcee promoted to reviewing; ready for owner promote.
```

epm:awaiting-promotion2026-05-14T00:38:12.970Z· agent

Clean-result dce3bcee-3dfa-44a9-b5bf-c52b7909ceab passed round-2 critic pair (claude pass, codex pass; round-1 reconcile…

Clean-result dce3bcee-3dfa-44a9-b5bf-c52b7909ceab passed round-2 critic pair (claude pass, codex pass; round-1 reconciler-required single-paragraph framing fix applied via PATCH). Experiment moved to awaiting_promotion; CR status flipped to reviewing for the dashboard promote queue.

Comments · 0

No comments yet. (Auth + comment composer land in step 5.)