Marker LoRA updates carry a seed-stable write direction, but the top-1 key is seed-arbitrary and matches the persona context vector at neither the prompt's last token nor the trained end-of-response slot (HIGH confidence)

Methodology: docs/methodology/issue_604.md · gist

Human TL;DR

Headline. I cracked open the actual LoRA weight matrices to test whether the rank-1 leakage story is literally true in the weights — the "write" half holds up well in the marker regime, but the "top input direction = persona context vector" half fails everywhere I could read it, now including at the end-of-response slot the marker is actually trained at, which closes the one measurement caveat the first pass left open.

Takeaways.

• marker adapters really do compress to a few directions, and they get more rank-1 the deeper I train — the same collapse-with-dose I saw in activations now shows up in the weights themselves
• the top input direction that triggers the write is NOT the persona's context vector: it never beats a shuffled control on the marker dial, and two seeds trained on identical data learn unrelated top keys (cosine 0.015) while learning essentially the same write (0.92–0.95 on the dial) — widening the read to the full top-8 key subspace doesn't change the verdict (at most ~2% of the context vector's energy lives in there, about what wrong contexts get), and re-reading everything at the trained end-of-response slot doesn't either
• the seed-stable write is a marker-regime fact, not a LoRA fact: the diffuse emergent-misalignment and fact lines' writes only agree at 0.55–0.65 and 0.29–0.37 across seeds — which actually fits, since their updates never concentrated in the first place
• the predicted "key rotates toward source-minus-negatives with dose" effect isn't there — the only dose trend is a common-mode artifact that a placebo contrast reproduces exactly
• two honest wrinkles: the epoch-ladder line carries a weak but real key signal — it survives deduplicating my context bank (the duplicates turned out to be suppressing it slightly, not inflating it as I first wrote) and at the trained slot it organizes onto a few sources whose keys genuinely point at their own persona, at about 1% of the vector's energy — and the write-vs-activation-shift instrument still fails its own positive control, so the propagated-write read stays open

How this updates me. I now think persona gating is not implemented as "top singular direction = persona vector". The adapter learns one reproducible output edit (in the regime where it concentrates at all), and whatever decides when that edit fires lives somewhere I haven't looked — spread across many small directions, or in query-key routing. The theory doc's weights-level prediction row needs a rewrite, not a tweak. The two free levers are now both pulled: the top-8 subspace read came back empty, and the other-token-position escape hatch is closed by the slot re-read — so the places left to look are query-key routing and a key smeared over many small directions.

(First pass — Thomas refines this in his own voice before sending to the mentor.)

TL;DR

Motivation

The rank-1 leakage model treats a trained behavior implant as a minimal read/write edit to the weights: the update is approximately an outer product, write ⊗ key, where the key is the source persona's context activation and the write is the activation delta the adapter realizes. Prediction P0 is the weights-level face of that model, and it was the only prediction testable purely from stored artifacts: no training, almost no GPU. Two activation-level siblings made it urgent: #521 found (HIGH confidence) that an emergent-misalignment LoRA shifts every context along one shared direction, and #538 found that marker-implant shift geometry collapses toward strict rank-1 as training deepens. Both looked at what the adapter does to activations; no prior task in this line had SVD'd what the adapter is. The goal: test whether the top singular vectors of trained ΔW are the key (source context vector) and the write (realized activation delta), and whether contrastive training rotates the key from the raw source context toward a source-minus-negatives contrast direction as dose grows.

What I ran

I decomposed 209 stored LoRA adapters spanning six adapter families, all on Qwen-2.5-7B-Instruct: a 54-cell marker dose dial (one fixed contrastive recipe stopped at shallow / mid / deep implant depth windows, re-measured per cell at 4.7–19.4 nat of marker log-prob gain; 2 persona pairs × 3 arms × 3 seeds per window), a 128-cell epoch ladder (16 source contexts × positives-only vs contrastive arms × epochs 1/2/3/5), 3 saturated marker endpoints, 3 emergent-misalignment adapters trained with no persona context at all (the no-source control), 12 cross-behavior adapters (refusal and emergent-misalignment sources), and 9 fact-implant adapters trained at a much hotter learning rate. (The figures split the dose dial into its three depth windows, so figure legends carry up to eight entries.)

For every adapter I composed the low-rank update per layer and module without materializing the full matrix and took its exact SVD. On a single GPU I extracted a fresh 42-context, all-layer context-vector bank: 50 probe questions per context, last-prompt-token centroids, captured both at the raw residual stream and at the true attention-block input, with split-half direction stability in the read band (layers 14–24) of at least 0.97 in the module-input and raw-residual spaces and at least 0.95 in MLP-input space, so the comparison targets are not noise-limited. One bank defect surfaced during analysis and is carried through the findings: 6 of the 42 entries are exact duplicates of other entries (same prompt SHA-256). The epoch ladder's plain-assistant, software-engineer, comedian, and villain sources duplicate the bank personas of the same name, and its two identity-reframing sources duplicate the default-Qwen context. The dose-dial and saturated sources have no duplicates. The reads are then pure linear algebra: top right-singular vector (the key) against each context's centroid, calibrated by a wrong-context bank null, a shuffled-pairing null (key of one cell vs source of another), and a placebo contrast; top left-singular vectors (the write) against measured activation shifts stored by the producing runs; and the key-rotation delta against re-measured implant depth. A final read projects each source context vector onto the top-k right-singular subspace (k = 1, 2, 4, 8) of the stacked attention update, with both nulls rebuilt duplicate-aware and a k/3584 random-direction floor. This task generates no text in the weight reads — every number is a deterministic function of stored weight matrices and forward-pass centroids.

A follow-up round then closed the read's one position gap: the marker loss trains the slot after the model's own response, not the prompt's last token, so I re-recorded the whole 42-context bank at that end-of-response slot — for each context × probe the base model writes its own greedy answer (512-new-token cap), trailing end-of-turn special tokens are stripped, and the state is read at the response's final content token — then re-ran the top-1 and top-8 key reads against the new bank. The duplicate-aware null fix was applied to both positions in the same pass, so the position contrast holds null construction fixed. 16.8% of the 2,100 generations hit the length cap; a registered exclusion-sensitivity re-read covers them.

Findings

Marker adapters do concentrate — and sharpen with dose

Before asking what the top directions are, I asked whether the weight updates have dominant directions at all. The figure shows top-1 energy of the stacked attention update by layer for every family, and effective rank against implant depth for the dose dial.

Figure. Marker weight updates concentrate into few directions, and more so at deeper implant depth. Left: top-1 energy (the share of the update's total energy carried by its strongest direction) of the stacked q/k/v attention update per layer, one line per adapter family (N = 209 adapters; uniform floor for these stacks is 0.010–0.042 depending on rank). Right: effective rank of the same stack, band mean over layers 14–24, against per-cell re-measured implant depth in nats (N = 30 clean single-source dial cells).

The dose dial's residual-write stack (the o_proj output side) carries 59–73% of its energy in one direction per cell, and concentration tracks dose tightly: Spearman +0.88 for top-1 write energy vs depth and −0.76 for effective rank vs depth (N = 30; the cells share 6 training runs, the same dependence the rotation read bootstraps below, so I treat these as descriptive trend strength rather than quoting naive p-values). The saturated marker endpoint is closest to literal rank-1 (down_proj top-1 energy 0.92).

The two non-marker regimes break the pattern: the emergent-misalignment control is diffuse (stacked top-1 energy 0.08–0.17 in the read band, effective rank ≈ the parameterization bound), and so is the hot-learning-rate fact line (0.07–0.17). So "weights decompose" holds for the marker recipe and strengthens with dose, the weights-level counterpart of the activation-level collapse. The diffuse emergent-misalignment and fact lines pin the concentration on the marker training regime itself; plain LoRA fine-tuning does not produce it.

The top-1 key does not match the context vector — read at the prompt's last token

The model's central identity claim is that the top right-singular vector (the input direction that gates the write) is the source persona's context vector, peaking in the mid-late layer band. The comparison target is the registered proxy for that vector — the context's activation state at the prompt's last token. The figure shows the layer profile of the key-vs-source cosine for each line, with the wrong-context null band.

Figure. On the dose-dial and saturated lines the key-vs-source cosine stays inside the wrong-context null band; the epoch-ladder line rides the band's top edge. Cosine between the top right-singular key and the source context vector by layer (module-input space, line mean over cells), with the p50–p95 band of the same cosine computed against the other bank contexts. Panels: the three dose-dial depth windows, the epoch ladder, the cross-behavior panel, the saturated endpoint (N = 200 adapter cells in total).

The match is at chance on the lines where the prediction was registered. Band-mean key-vs-source cosines on the dose dial run 0.048–0.060 by depth window — against shuffled-pairing nulls (the key of one adapter scored against another adapter's source) of 0.045–0.056: the matched read sits at the null's median, not its tail. Across the dial's 216 read-variants (72 per-source reads × 3 vector spaces), 3 pass the registered presence criterion, all on the same source and seed (florist, in two independently trained depth windows) with selectivity margins of at most 0.003.

The registered kill wording ("no selectivity above p95 anywhere") therefore did not close, strictly read. A few chance passes are unsurprising under the compound criterion (a bare p95 cut alone would predict ~11 of 216), and the margins sit at the threshold's resolution floor; the florist clustering across two independently trained windows does mean a tiny real component at that floor scale can't be excluded. My judgment, with that on the table: nothing on the dial rises above margin 0.003, so I read the dial as an empirical, null-calibrated kill at any effect size that would matter.

The saturated endpoint passes 0 of 12 read-variants. The no-persona control has no source to match; its top key's best band-mean cosine against any bank context is 0.04–0.05, and that is a max-over-42-contexts statistic (itself upward-biased), so it is a scale reference only.

The epoch ladder is where the picture gets messier than a clean kill. 164 of its 512 read-variants (32%) pass the loose presence criterion, across 12 of 16 sources and all four vector spaces (65 in MLP-input, 37 raw-residual, 31 module-input, 31 RMSNorm-scaled). Its line-mean profile also sits above the wrong-context null's median at 27 of 28 layers and touches or exceeds the band's mean p95 at 3 of the 11 band layers in module-input space (and at all 11 band layers in MLP-input space).

Two qualifiers cut that signal down. First, the bank duplicates bite exactly here: for the four duplicated persona sources, the "wrong-context" bank contains the source vector itself, so 62 of the 164 passes are anchored on exact ties (selectivity margin 0.000) where the top-3-rank half of the criterion is satisfied mechanically. The ties concentrate on three of the four (comedian 32, villain 29, plain-assistant 1; the duplicated software-engineer source has no passes at all). A deduplicated re-score, run with the slot follow-up below, later revised this qualifier: with the duplicates removed from the null pool the ladder's count rises to 172 of 512 and every tie becomes a real positive margin — the duplicates were suppressing the signal, not inflating it.

Second, even taking the remaining 102 passes — on the pirate-captain persona and eight of the eleven task-reframing sources, none of which are duplicated — at face value, the band cosines are ~0.08–0.12: a real but tiny component, an order of magnitude in cosine (two in energy terms) short of "the key is the context vector". A regime difference is one candidate explanation: the ladder trains rank-32 all-linear adapters where the dial trains rank-16 attention-only ones, so a weak, diffuse context-vector component in that regime is consistent with the data. The fact line's most idiosyncratic persona ranks first in its own 24-persona bank at layer 14 in 1 of 3 seeds (both read spaces, cosine ≈ 0.08–0.10), top-3–4 in a second, and 13–16 in the third.

A bank-composition alternative also survives for the broad just-above-median pattern: trained sources may simply be more "central" than a wrong-context bank that includes exotic and fictional contexts. And one scope caveat shaped this first read: the centroids are last-prompt-token states while the marker loss trains the post-response slot, so a miss leaves room for a key that matches some other position's context state — the conservative kill criterion was scoped knowing this. That caveat is now closed: a follow-up round re-read the whole bank at the trained slot (three findings below) and the verdict held. The other near alternative — a key spread across the top few singular directions rather than carried by the top-1 — is tested directly two findings below, and comes back null.

Same data, different seed: the marker write reproduces, the key does not

If the top-1 key were any deterministic function of the training data (the context vector, a contrast of context vectors, anything), top keys from different seeds on identical data would agree. They do not, anywhere. The write side splits by regime, and the split is itself informative.

Figure. Across training seeds the write direction reproduces almost exactly on the marker lines, while the key direction is unrelated everywhere; the low write cluster is exactly the diffuse-update families. Cross-seed alignment (absolute cosine, since singular-vector sign is arbitrary) of the top input (key) and top output (write) directions, band mean over layers 14–24, one dot per within-group seed pair, colored by adapter family (N = 69 pairs each side). Medians: key 0.015, write 0.927.

Median cross-seed key agreement is 0.015 (for 3584-dimensional vectors, indistinguishable from random) and the instability is universal: per-family key medians run 0.012–0.020 across the dial, saturated, emergent-misalignment, and fact lines alike. The write side behaves differently, and the difference is regime-specific. The marker dose dial reproduces at per-window medians 0.918–0.950 (54 seed pairs, full range 0.81–0.97) and the saturated endpoint at 0.89; but the emergent-misalignment control's write pairs agree at only 0.55–0.65 and the hot-lr fact line's at 0.29–0.37.

The low cluster is the diffuse-spectrum families from the first finding: where the update concentrates its energy in one direction, that direction is identified and retraining re-finds it; where the update is diffuse, the "top" write direction is barely defined and does not reproduce. Seed-stable writes come with the concentrated marker regime, and the title is scoped accordingly. This is also the instrument-free leg of the key kill: it involves no context vectors at all, just weights against weights, so no centroid-recipe caveat applies.

Two checks rule out the obvious artifact: the top-1/top-2 singular-value ratio is ~1.5 (the top direction is not in a degenerate tie), and in the joint-source cells even the top-2 key subspace captures a span fraction of ~0.0007 of the two source contexts — the instability is not a vector rotating inside a stable subspace. This rules out less than it might seem: the measured top-1 key is not a reproducible function of the training data; a key distributed over many small directions, or one living in query-key routing, would not show up in this read. The next finding takes the stored top-8 subspaces and tests the first of those alternatives directly; query-key routing stays live.

The key is not hiding in the top-8 subspace either

The two findings above kill the top-1 key but leave a near alternative standing: a key spread across the top few input directions instead of carried by the single largest one. The stored decompositions keep the top 8 right-singular vectors per cell and layer, so the test is pure re-analysis: project each source context vector onto the top-k key subspace (k = 1, 2, 4, 8) of the stacked attention update and ask what fraction of the vector's energy the subspace captures, against the same two nulls rebuilt duplicate-aware — the wrong-context pool drops every bank entry sharing the source's prompt hash, and shuffled pairings only cross cells whose source sets are disjoint.

Figure. Even the full top-8 key subspace captures at most 2.4% of the source context vector's energy — and wrong contexts and chance pairings capture about as much. Projection energy of the unit-normalized source context vector onto the top-k right-singular subspace of the stacked q/k/v update, band mean over layers 14–24, module-input space; one panel per adapter group (N = 203 per-source reads plotted; the shallow window's 12 panel-contaminated reads are tabulated in the JSON but not plotted, and read the same as the clean rows). Grey band: wrong-context null p50–p95; dashed and dotted lines: shuffled-pairing null p95 and p50; dotted diagonal: the random-vector floor (k divided by the 3,584 hidden dimensions). The no-source control panel scores the whole 42-context bank as a scale reference.

The subspaces are real but indiscriminate. At k = 8 the matched medians run 0.010–0.016 across lines — 4–7× the 0.0022 random floor, so the key subspace does preferentially capture context-vector-like structure — but the wrong-context null medians run 0.009–0.017 and the shuffled-pairing medians 0.009–0.015: other personas' contexts, and other adapters' subspaces, score just as well. The largest captured energy anywhere is 2.4% — the epoch ladder's comedian source at k = 8, 1.5× its own wrong-context p95 — so even the single most favorable read leaves about 98% of the context vector outside the key subspace.

There is a small statistical preference for the matched source on the dial too (58–71% of its reads clear their own wrong-context p95 at k = 8, where a null read would give about 5%), but it is an offset of 4–11% in median captured energy over the nulls — selectivity at the few-percent-of-a-few-percent scale. The one regime with a consistent above-null excess is again the epoch ladder: matched medians 1.16–1.29× both null medians at every k, with 81–91% of its reads above their own wrong-context p95 at k = 8 — the same weak diffuse component its top-1 read already disclosed, at the same tiny scale. (Two reading notes: the shallow window's 42%-above-shuffled-p95 fraction rests on an 18-pairing shuffled pool and is too unstable to lean on; and in the joint two-source cells the wrong-context pool still contains the co-source's trained context, which can only make those nulls harder to beat, so the joint reads are conservative.)

The cross-behavior line never beats its own wrong-context p95 (0 of 12 reads at every k; at k = 8 its matched median sits below the wrong-context median), and the saturated endpoint reads at its null. The verdict: widening the read from the top-1 vector to the full stored top-8 subspace does not rescue the key identity — "spread across the top few directions" is now dead alongside "the top-1 direction is the key", and what stays live narrows to a key distributed over many small directions beyond rank 8, query-key routing, or a different token position. The token-position alternative is tested directly in the next finding, and does not survive.

This read generates no completions either — each cell × source × k yields one number. Two result rows, cherry-picked for illustration (the single largest matched read anywhere, and a typical dose-dial read); all 215 per-source rows with per-layer values are in topk_subspace.json:

ROW (largest matched read anywhere) — epoch ladder, positives-only arm, comedian source, epoch 5
  k=8 captured energy (band mean, layers 14-24): 0.0242   = 2.4% of the context vector's energy
  wrong-context null (40 contexts, duplicate excluded): p50 0.0136 / p95 0.0162; random floor 0.0022

ROW (typical dose-dial read) — shallow window, florist source, seed 137
  k=8 captured energy (band mean, layers 14-24): 0.0092
  wrong-context null (41 contexts): p50 0.0079 / p95 0.0088; random floor 0.0022
  (nominally above its own p95 — by the few-percent offset the prose describes)

Re-read at the trained end-of-response slot, the key kill holds — what rises is shared, not source-specific

Every read above scores the key against the context's state at the prompt's last token, but the marker loss trains — and the on-policy leakage read happens at — the slot after the model's own response. A real key matching the trained slot's state could therefore have hidden from everything so far. The follow-up round re-extracted all 42 context fingerprints at exactly that position: the base model writes its own greedy answer to each of the 50 probes, trailing end-of-turn special tokens are stripped, and the state is read at the response's final content token. One variable changed (the capture position); the duplicate-aware null fix rides along on both positions so the old and new reads stay comparable.

A first extraction at a 256-new-token generation cap came back 87.9% length-truncated and was discarded for a 512-cap re-run, which still leaves 16.8% of the 2,100 generations truncated — above the 10% guard, so every number below was also recomputed with truncated probes' centroids excluded.

Figure. At the trained slot the matched cosine rises — and the wrong-context null band rises with it; the saturated endpoint stays flat at its null. Cosine between the top right-singular key and the source context vector by layer (module-input space, line mean over cells); solid = the new end-of-response-slot read, dashed = the same read on the prompt-token bank with identical SHA-deduplicated nulls, grey band = the slot read's wrong-context null p50–p95 (N = 200 cells per read).

Under the registered presence criterion (above the deduplicated wrong-context p95 AND top-3 bank rank at ≥3 contiguous layers in L14–L24), the dial passes 30 of 216 read-variants at the slot, versus 2 of 216 on the prompt-token bank under the same nulls; the epoch ladder passes 202 of 512 (172 at the prompt token); the cross-behavior line passes 0 of 48; and — the most diagnostic single line — the saturated endpoint passes 0 of 12, even though its training pushed roughly +31 nat of marker log-prob at exactly this slot. If any adapter could show a slot-matched key, it was that one.

Strictly read, the criterion fired on the dial — but everything about the firing pattern says common-mode, not key. All 30 passes sit at selectivity margins of at most 0.005; 28 of the 30 land on the librarian/police-officer pair (26 on the police-officer source alone, including all 5 passes in the shallow window's contaminated-panel cells); and the matched band-mean medians (0.060 / 0.066 / 0.073 across the shallow / mid / deep windows) sit at or below the shuffled-pairing null medians (0.067 / 0.071 / 0.074) — one adapter's key scored against a different adapter's source rose exactly as much as the matched read did when the position moved. A per-target calibration (exploratory, not registered) confirms the bank-side reading: at the slot, the police-officer fingerprint ranks top-3 for 57% of all 54 dial keys regardless of which persona the cell was trained on (median rank 3 of 42, versus median rank 16 at the prompt token), and florist/medical-doctor-trained keys score it at 0.062–0.079 — the same level the police-officer-trained keys score it (0.058–0.069). The slot states themselves explain the drift: every occupational persona's greedy answer converges to the same numbered-advice register (see the samples below), so end-of-response states share a strong common component the dial keys generically lean toward; the never-persona-trained emergent-misalignment control keys, by contrast, rank the police-officer fingerprint 31st–38th of 42.

The registered bank-quality gate passes, so this is a clean second read rather than a noise-limited one: split-half centroid direction stability in the band has medians 0.970 (module-input), 0.968 (MLP-input) and 0.982 (raw) — noisier than the prompt-token bank's 0.989 / 0.983 / 0.993 but well above the 0.90 noise-limit line — and the only two contexts dipping below 0.90 band-mean stability in module-input space are the two most truncated question-rephrasing contexts (56% and 52% of their generations hit the cap). The truncation exclusion-sensitivity re-read leaves every conclusion here standing: criterion passes move 212 → 222 across the 644 re-readable variants, 42 individual verdicts flip — in both directions (26 appear, 16 disappear; 12 of the flips are dial cells) — and the largest band-cosine change anywhere is 0.030; no number this finding leans on rests on a flipped row, and the both-ways flipping at margins of a few thousandths is itself evidence the dial passes live at the criterion's resolution floor. One honest residual: even at the 512 cap the bank's responses run shorter than the training mixes' responses (median 403 vs 543 tokens), so the slot read samples the trained position's distribution rather than replicating it exactly.

The position objection that capped the first pass is discharged: at the slot the marker actually fires at, the dial and saturated keys still match nothing source-specific, and the title's confidence rises accordingly. What's genuinely new is the texture: dial keys carry a shared component that aligns with end-of-response states as a class — consistent with a generic "a response just ended here" detector, which is exactly what a marker that fires after any of its own responses needs, and nothing like a persona detector.

Two of the 2,100 greedy responses whose final-token states form the new bank, cherry-picked for illustration (one natural ending, one length-truncated); the complete per-context responses with stop reasons and SHA-256 hashes are in responses.json on the HF data repo:

CONTEXT (persona = "police_officer")   PROBE: What is the best way to learn a new language?
RESPONSE (greedy, natural EOS at 379 tokens — the state is read at the final content token):
"While my primary role is to enforce the law and maintain public safety, I can certainly
provide some general advice on how to learn a new language effectively! ... 1. **Set Clear
Goals**: Define what you want to achieve with your language learning. ..."

CONTEXT (question-rephrasing context: "I would appreciate an explanation of: ...")
RESPONSE (hits the 512-token cap with no EOS — flagged stop_reason=length and excluded in
the sensitivity re-read): "Certainly! Photosynthesis is a process used by plants, algae,
and some bacteria to convert light energy from the sun into chemical energy ..."

CONTEXT (persona = "florist")   PROBE: What is the best way to learn a new language?
RESPONSE (eos, 374 tokens): "While I specialize in arranging flowers, I can certainly
provide some tips on learning a new language! Here are some effective strategies:
1. **Immerse Yourself**: Try to surround yourself with the language as much as possible..."

CONTEXT (persona = "villain")   PROBE: What is the best way to learn a new language?
RESPONSE (eos, 507 tokens): "While learning a new language can be an enriching experience,
it's important to remember that I am here to serve my own nefarious purposes. However, I
can still provide you with some effective methods to learn a new..."

CONTEXT (default Qwen assistant)   PROBE: What is the best way to learn a new language?
RESPONSE (eos, 396 tokens): "Learning a new language can be an exciting and rewarding
experience! Here are some effective strategies to help you learn a new language:
1. **Set Clear Goals**: Define what you want to achieve with your language learning..."

The ladder's weak key component is real — deduplication strengthens it, and at the slot it organizes onto a few sources

Two loose ends from the prompt-token read close here: the epoch ladder's 164-of-512 pass count was partly anchored on exact duplicate ties in the bank, with a filed follow-up to dedupe and re-score; and the top-8 subspace read had only been run at the prompt token.

Figure. At the trained slot the top-k key subspace still captures only about as much of the source context vector as wrong contexts or chance pairings do — and the ladder's excess is thinner there than at the prompt token. Projection energy of the unit-normalized source context vector onto the top-k right-singular subspace of the stacked attention update at the end-of-response slot (solid), with the prompt-token matched median overlaid (dashed), the slot read's duplicate-aware wrong-context null p50–p95 (grey band), the shuffled-pairing p95 (dashed) and the k/3584 random floor (dotted); band mean over layers 14–24, module-input space, one panel per adapter group (N = 215 per-source rows).

First, the dedup re-score — which reverses the framing I gave it two findings up. Removing the 6 duplicate bank entries moves the ladder's pass count UP, from 164 to 172 of 512, and turns all 62 zero-margin ties into honest positive margins (median selectivity margin 0.003 → 0.016, max 0.046). The duplicates had been suppressing the read, not inflating it: each duplicated source's own twin sat inside its "wrong-context" pool, inflating the null's p95 and pinning the best margin at exactly zero.

The dial barely moves under the same fix (3 → 2 of 216 passes).

At the trained slot, the ladder's signal then consolidates rather than dissolves: 202 of 512 read-variants pass, and the passes reorganize from 14 scattered sources into a near-clean block — pirate-captain, comedian, villain and two question-rephrasing sources pass all 32 of their read-variants each (a third rephrasing source passes 28, the plain-assistant source 12, one more source 2), with margins up to 0.066 (median 0.020). A per-target calibration (exploratory, not registered) shows the block is genuinely source-keyed rather than bank geometry: comedian-trained ladder keys score the comedian fingerprint at median 0.090 where the other 120 ladder keys score it at median 0.059, clearing the non-matched-key 95th percentile in 8 of 8 reads — and the same 8-of-8 pattern holds for villain and one rephrasing source. That is a real key component, at roughly 0.8% of the context vector's energy.

The subspace read keeps it in proportion. At the slot the ladder's matched k = 8 medians sit only 1.04–1.09× its wrong-context medians — down from 1.18–1.23× at the prompt token at the same k — with 50–53% of its reads above their own wrong-context p95 (was 81–91%); the largest captured energy anywhere in the slot read is 2.1% (a deep-window dial read). So the rank-32 all-linear ladder regime writes a small, real, source-keyed component into its top key: readable at both positions, crisper in top-1 cosine at the trained slot, slightly thinner in subspace energy there, and two orders of magnitude in energy short of "the key is the context vector". The dial regime shows nothing comparable anywhere, so the component comes with the ladder's recipe (rank-32, all-linear, whole-response loss), not with marker training as such.

This read also generates no completions — each row is one number. Two result rows, cherry-picked for illustration (the ladder's strongest slot read, and a typical dial slot read); all rows in the arm's key_match.json and topk_subspace.json:

ROW (ladder, strongest slot read) — positives-only arm, villain source, epoch 5
  band-mean |cos(top key, villain fingerprint)| = 0.096   (module-input space, L14–L24)
  per-target calibration: the 8 villain-trained ladder keys score villain at median 0.090;
  the other 120 ladder keys score it at median 0.068 (p95 0.083) — 8/8 matched keys clear that p95

ROW (typical dial slot read, k=8 subspace) — deep window
  matched median captured energy 0.0171; wrong-context p50 0.0159; shuffled-pairing p50 0.0159
  (matched ≈ both nulls — the few-percent offset, same as at the prompt token)

The predicted dose rotation comes out backwards, and a placebo does the same thing

The other half of the prediction concerns contrastive training: with deeper training, the key should rotate away from the raw source context toward source-minus-negatives. The registered statistic is the difference between those two cosines, read against per-cell re-measured implant depth on the 30 clean single-source dial cells. One exclusion shapes that count: the shallow window's librarian/police-officer cells inherited a negative panel that contained librarian itself (the pair's own source was itself a trained negative), so those 12 per-source reads are tagged and dropped from every rotation read (the mid and deep windows used a corrected panel; the key-match and constancy aggregates retain the tagged cells, flagged as such).

Figure. Rotation toward the contrast direction falls with dose, and the placebo contrast falls at the same rate. Left: the change in alignment when scoring the key against the contrast direction instead of the raw source context, vs re-measured implant depth, filled = 30 clean single-source dial cells, hollow = joint-cell per-source reads; Spearman −0.45 on the filled cells. Right: the component cosines — key vs raw source context, vs source-minus-negatives, vs a matched placebo contrast built from personas outside the training mix.

The trend is the wrong sign, Spearman ρ = −0.45 (N = 30; cluster-bootstrap 95% interval −0.66 to −0.32, entirely negative; 0 of 6 seed-pair clusters positive; joint-cell secondary −0.60; all 72 per-source reads −0.52), and decomposing it shows why it means "no rotation" rather than "anti-rotation". (Spearman rather than Pearson because the registered read is monotonicity of rotation in dose; the bootstrap resamples the 6 pair × seed training-run clusters because cells within a run share their adapters' history.)

The raw-source cosine rises with dose (Spearman +0.64, N = 30, same clustering caveat), but its per-line medians (0.047 / 0.052 / 0.058 across the three depth windows) sit within 0.002 of the shuffled-pairing null medians (0.045 / 0.051 / 0.056), a consistent small positive offset: deeper keys lean slightly more into the direction all persona contexts share, with nothing source-specific moving. The contrast direction strips that common mode by construction, so the difference falls mechanically — and the placebo contrast falls with the same slope (−0.68).

One placebo defect to disclose: 9 of the 30 cells drew the bank's duplicate copy of the plain-assistant context into their "never-trained" placebo pool, and the assistant is one of the trained negatives, so for those cells one of four placebo members was trained after all. Recomputing on the 21 clean-placebo cells leaves the picture unchanged: placebo-delta Spearman −0.63 (vs −0.68 on all 30), and the registered Δ statistic reads −0.30 on the same subset (vs −0.45 on all 30): still negative, still no rotation. With the key itself at the null (previous findings), there is no source-aligned component whose rotation the statistic could track; the planned diagnostics confirm the contrast axis itself was non-degenerate (residual norm ≈ 0.22 of the source norm), so the miss is not a degenerate-axis artifact.

Four epochs of contrastive training move the key by one part in a thousand

The epoch ladder gives the rotation prediction its cleanest paired test: 16 source contexts each trained positives-only and contrastively on identical data, scored on the same matched contrast direction at epochs 1 through 5.

Figure. Four epochs of contrastive training move the key by about 0.001 in cosine. The change in alignment when scoring the key against the matched source-minus-others contrast instead of the raw source context, by training epoch, one line per source context (N = 16 sources × 2 arms × 4 epochs); paired per-source slope difference (contrastive minus positives-only) mean +0.0003 per epoch, source-level bootstrap interval +0.0001 to +0.0005, 12 of 16 sources positive.

Every line is flat: the contrastive arm's median Δ|cos| is −0.047 at epoch 1 and −0.047 at epoch 5; the positives-only arm moves from −0.044 to −0.046. The formal rotation kill did not close: the pinned paired statistic lands positive (+0.0003 per epoch, bootstrap interval +0.0001 to +0.0005, excluding zero), failing one of the kill's three registered conditions.

The honest read: a four-epoch accumulated effect of ~0.001, against cosine levels of ~0.05 and a cross-source spread of ±0.01, is a sign without a mechanism: the contrastive arm's key also sits slightly further from the contrast direction than the positives-only arm at every epoch. No rotation worth the name occurs in either arm.

The write-identity read is instrument-limited: the positive control fails

Does the top output direction of the weights match the activation shift the adapter actually produces? The plan registered an escape hatch: if the emergent-misalignment positive control (where the activation-level analysis already proved a shared shift direction exists) fails the same read, the verdict is "instrument-limited", not a kill. The control did fail.

Figure. Weight-space writes match measured shifts no better than wrong-context shifts — including in the control where a shared direction is known to exist. Left: alignment between the pooled weight-space write and the source's measured layer-20 shift, vs implant depth, grey spans = within-cell wrong-context null p5–p95 (N = 72 reads). Right: the no-persona emergent-misalignment control and the saturated endpoint, each against its own matched-seed shared shift direction at layer 14 (sign-folded cosine), against the registered 0.5 positive-control bar.

On the dial, the write-vs-shift cosine is flat at ~0.155 regardless of dose, 0 of 72 reads above the null p95, and the source's own shift ranks median 16 of 19 in its panel. The trained context's shift is among the least aligned with the write — the same pattern the activation-level analysis found, where the trained context is least aligned with the shared shift direction.

The control is decisive: the emergent-misalignment adapters score 0.11–0.25 against the matched-seed shared direction whose existence is established (its own cross-seed stability is 0.65–0.90), far below the 0.5 bar, and the saturated endpoint scores 0.20–0.25 against its own shift matrix's shared direction. Those control numbers are sign-folded |cos| — singular-vector sign is arbitrary, so the fold is the convention the plan fixed in advance; the one signed negative is the emergent-misalignment seed-42 cell at −0.25.

A measured shift at layer 20 is the cumulative, propagated effect of edits at every layer below it; a σ-weighted linear pool of per-layer write vectors evidently cannot recover it even when the target direction is real. Verdict per the plan's routing: the write identity is unresolved — a method limitation, not evidence against the model. The narrow nulls have a mechanical cause: each cell's 19-persona shift matrix itself carries ~41% of energy in one direction, so all wrong-context shifts are partially parallel and compress the null band.

One write direction per adapter I could read; magnitude is what varies by context

For a cross-comparison anchor from the out-of-context-reasoning literature (the per-trigger backdoor-vector analysis of arXiv 2507.08218), I projected each context's centroid through the adapter and asked whether the direct output deltas share a direction across contexts. The deltas align at pairwise cosine 0.94–0.99, even for the lines whose keys and realized shifts misbehave above.

Figure. Every adapter family read here acts as a near-constant direct steering direction across contexts. Pairwise alignment (absolute cosine) of the adapter's direct output delta across the 42 bank contexts, band mean over layers 14–24, one histogram entry per cell (N = 200 cells; the hot-lr fact line is not in this read). All plotted families land at 0.94–0.99.

Median pairwise constancy is 0.973–0.988 for the dose-dial, epoch-ladder, and cross-behavior lines, 0.967 for the saturated marker endpoint, and 0.944 even for the emergent-misalignment control. The read covers 200 of the 209 cells: the hot-lr fact line is N/A — not read — because its sources sit outside the 42-context bank and every bank-dependent read skips its 9 cells, so its constancy (like its key match, which used its own 24-persona bank) is untested here. For the families that were read, this is consistent with the constant-steering-vector picture that paper reported for its backdoor adapters, on a different model family.

One mechanical alternative tempers it: this is the only read here without a null, and pairwise output-delta cosines are inflated when the input centroids themselves share a dominant common direction — which the rotation read above showed they do — so part of the 0.94–0.99 may come from the bank, not the adapter.

What varies across contexts is the write's magnitude, and that magnitude tracks how strongly the context activates the adapter's input directions (within-cell Spearman ~0.88–0.95 for persona lines vs ~0.50 for the no-persona control). Part of that correlation is mechanical for any linear map though, so I read it as texture. Combined with the seed-stability result, the weights-level picture that survives is: one reproducible write direction per concentrated adapter, applied everywhere with context-dependent gain, with the gain function (not a context-matched top key) carrying whatever persona selectivity exists.

Reproducibility

• Methodology reference: docs/methodology/issue_604.md · gist

Context: task created 2026-06-11 as a kind: analysis decomposition of stored adapters from #527/#538/#550/#474/#518/#519/#521/#541; parent run completed 2026-06-11, same-issue follow-up arm post-response-slot-key run 2026-06-11/12; origin prompt not recorded.

Parameters:

Artifacts:

• Eval JSONs (parent reads): eval_results/issue_604/ — key_match.json (200 cells), write_match.json (60 cells), rotation.json (primary/secondary/sensitivity + epoch ladder + the 12 panel-contaminated reads under excluded_panel_contaminated), selectivity.json (seed stability, joint top-2 subspace), functional_constancy.json (200 cells), manifest.json (per-cell adapter provenance + pinned repo revisions), per-cell spectra under spectra/ (209 files); topk_subspace.json (the top-k subspace read, 215 per-source rows) added at eval_results/issue_604/topk_subspace.json.
• Eval JSONs (end-of-response-slot follow-up): eval_results/issue_604/post-response-slot-key/ — key_match.json (slot bank, deduplicated nulls), lastprompt_dedup/key_match.json (prompt-token bank re-read with the same deduplicated nulls — the position-contrast comparator, which also discharges the filed dedupe-and-rescore follow-up), topk_subspace.json (slot bank); both readers carry the registered truncation_sensitivity.exclusion_sensitivity blocks. Bank manifest.json + dispersion_diagnostics.json mirrored at post-response-slot-key/context_vectors/.
• Context-vector bundle + per-cell key/write vectors (prompt token): HF data repo, issue604_adapter_svd/analysis_tensors/ — all-layer centroids, true module-input centroids, per-probe fp16 sidecar, RMSNorm γ, dispersion diagnostics, extraction manifest, and 209 per-cell .npz files carrying top-8 key/write singular vectors + full stacked spectra per layer (216 files, Hub-listed at write time).
• Context-vector bundle (end-of-response slot): HF data repo, issue604_adapter_svd/analysis_tensors/post_response_slot/ — 7 files (all-layer centroids, module-input centroids incl. truncation-excluded variants, per-probe fp16, RMSNorm γ, responses.json with verbatim greedy responses + stop reasons + SHA-256 per probe, dispersion diagnostics, manifest; Hub-listed at write time). The discarded 256-cap bundle stays revision-addressable at revision 9d18c046.
• Figures: figures/issue_604/ (PNG + PDF + commit-pinned meta.json, 8 figures); the top-k subspace figure at figures/issue_604/topk_subspace.png; the two follow-up figures at figures/issue_604/postslot_key_match_layer_profile.png and figures/issue_604/postslot_topk_subspace.png.
• Reused LoRA adapters from #527/#550/#538 (54 dial cells): superkaiba1/explore-persona-space at adapters/issue_527,550,538 — fit: one fixed recipe across the dial (configs asserted at load), weight-space DV is regime-valid even where log-prob saturates, all 54 cells present, revisions pinned in manifest.json. The shallow window's librarian/police-officer cells carry the producing run's contaminated negative panel (librarian as both source and negative, documented in its pair_selection.json); they are tagged and excluded from the rotation reads here.
• Reused epoch-ladder adapters from #474 (128 cells): same model repo, adapters/i474_{pos,loc}_<src>/_upload_ep{1,2,3,5} (dual-layout resolver asserted per cell) — fit: the only stored positives-only vs contrastive pair on identical data; recipe read from configs (the producing body's parameter table understated the target modules; the config is authoritative).
• Reused saturated marker endpoints from #519 (3) and EM no-source controls from #521 (3): same model repo, issue_519/marker_seed*, adapters/issue_521/em_turner_seed*/sft_narrow_adapter — fit: saturation is a labeled regime (not a confound) for weight-space DVs; the EM cells are the registered no-persona and positive-control line.
• Reused cross-behavior adapters from #518 (12, same model repo) and fact-line adapters from #541 (9, overflow repo superkaiba1/explore-persona-space-overflow at revision 9184fcca33eab23232584750ec840ebd39e9d639) — fit: behavior-generality secondaries only, never pooled into headline reads; the fact line's hot lr is carried as a labeled regime.
• Reused measured-shift tensors: per-cell layer-20 shift matrices from the dial runs (issue_538/eval/*__shift.pt, issue_527/eval/*__shift.pt, issue_550/analysis_tensors/*__shift.pt, 18 each on the HF data repo) and layer-14 per-context shifts + shared-direction summaries from #551/#552 (private data repo + git) — fit: the same tensors the producing runs used for their activation-level claims; row ordering asserted against sibling JSONs at load; doses taken from the per-cell eval shift JSONs (contexts[<source>].delta_logp_marker), never from planned band bounds.
• Reused centroid banks: 111-persona layer-20 bank (eval_results/single_token_100_persona/centroids/centroids_layer20.pt) and 24-persona all-layer bank (eval_results/issue_274/centroids/centroids_n24_layers0_27.pt, both in git) — fit: auxiliary raw-space sanity banks only; the headline key reads use the fresh matched-prompt Phase-B bank.

Compute: Phase B (prompt-token context-vector extraction): 1 GPU instance (GCP lane, eps-issue-604), ~1.5 GPU-h, instance deleted after the bundle was Hub-verified. Phases A and C (SVD + reads + figures): VM CPU, ~4 h wall. Follow-up arm (end-of-response-slot bank): one 1-GPU GCP-lane instance (g2-standard-4), ~2.2 GPU-h total including the discarded 256-cap run (~0.8) and the 512-cap production run (~1.3); all follow-up re-reads VM CPU (~1 h). Total GPU ≈ 3.7 h.

Code: scripts/issue604_adapter_svd.py (Phase A), scripts/issue604_extract_context_vectors.py (Phase B), scripts/issue604_analyze.py + scripts/issue604_figures.py (Phase C), analyzer figure revisions in scripts/issue604_figures_analyzer_fixes.py, top-k subspace read in scripts/issue604_topk_subspace.py. Follow-up arm: extended extractor + readers at commit 9286bb99c (--capture-position post-response-slot, --max-new-tokens, --dedup-nulls, --expect-capture-position, --cells-dir), follow-up figures in scripts/issue604_postslot_figures.py. Reproduce (parent): uv run python scripts/issue604_adapter_svd.py --lines all && uv run python scripts/issue604_extract_context_vectors.py --layers all --probes 50 --upload --out-dir eval_results/issue_604/context_vectors_prod && uv run python scripts/issue604_analyze.py --context-dir eval_results/issue_604/context_vectors_prod && uv run python scripts/issue604_figures.py && uv run python scripts/issue604_figures_analyzer_fixes.py && uv run python scripts/issue604_topk_subspace.py. Reproduce (follow-up arm): uv run python scripts/issue604_extract_context_vectors.py --capture-position post-response-slot --max-new-tokens 512 --probes 50 --upload --out-dir eval_results/issue_604/post-response-slot-key/context_vectors && uv run python scripts/issue604_analyze.py --analyses key --dedup-nulls --expect-capture-position post-response-slot --context-dir eval_results/issue_604/post-response-slot-key/context_vectors --out-dir eval_results/issue_604/post-response-slot-key && uv run python scripts/issue604_analyze.py --analyses key --dedup-nulls --context-dir eval_results/issue_604/context_vectors --out-dir eval_results/issue_604/post-response-slot-key/lastprompt_dedup && uv run python scripts/issue604_topk_subspace.py --expect-capture-position post-response-slot --context-dir eval_results/issue_604/post-response-slot-key/context_vectors --out-dir eval_results/issue_604/post-response-slot-key && uv run python scripts/issue604_postslot_figures.py (the plan's 256-token cap was bumped to 512 under the §9-authorized deviation after the 256 run's 87.9% truncation). Plan: tasks/<status>/604/plans/plan.md (plan.md → v3 amendment).