Rank-1 LoRA marker implants behave like ungated writes at the band-stop dose (MODERATE confidence)

Methodology: docs/methodology/issue_621.md · gist

Human TL;DR

Headline. Pushed the marker implant down to rank 1 (one (a, b) outer product per module, no SVD) and stopped training the moment source log P(marker) enters the [5,12] nat band. The read vector barely rotates — cos(a_trained, a_init) ≥ 0.988 across all 30 cells — and the rotation half of the joint threshold passes 23/30 overall (read placement 5/12, write 12/12, bridge 6/6). The 7 read-arm misses sit 0.1–0.7 percentage points above the 0.15 cap, but the persisted a(t) ladder confirms all 7 had plateaued by termination (last-30-step |Δcos| ≤ 1.9e-3, ~6e-5 per step), so the read-arm 5/12 IS the asymptote at this dose, not a transient still rotating. The implant looks like an ungated write at this dose, and "a · v_persona" doesn't rank-order who leaks — plain cos(v_persona, v_source) from the same bank with no adapter beats it in 0/30 cells. But because a barely moves, the firing predictor reduces to a near-random projection × a learned scalar, so the firing-vs-leakage comparison is structurally degenerate at this dose, not falsifying.

Takeaways.

• The A-init control #604 couldn't run lands cleanly at this dose: across 30 cells the trained A is within cos = 0.988–0.998 (median 0.995) of its random Kaiming start, with median angular drift ≈ 5.9° and read-arm minimum ≈ 8.9°. Whatever does the persona selection lives in B + the base prior, not in a learned read direction.
• The joint rotation criterion (|cos| > 0.9 AND Δa/a₀ < 0.15) passes in 23/30 overall, but the breakdown matters: write placement 12/12, bridge placement 6/6, read placement 5/12 — the 7 read-arm misses are the loss-exposed placement, all sitting 0.1–0.7 pp above the cap. The 10-step checkpoint ladder (now run) shows all 7 had plateaued at termination (verdict=plateau, tail slope ~6e-5 per step, well below the 1e-4 heuristic), so 5/12 IS the read-arm asymptote at this dose, not a transient.
• The write direction is consistent with the marker, directionally as #604 saw at rank 16: cos(b, W_U[marker]) reaches median 0.79 on down_proj and 0.63 on o_proj (max over layers 20–27), vs a frequency-matched null at 0.05–0.09 and random floor 0.017.
• The firing predictor s·(a·v_persona)·||b|| loses to plain bank geometry in every one of 30 cells (median Δρ = −0.33 read placement, −0.40 write, −0.25 bridge). 3/12 read-arm cells cross the ρ ≥ 0.5 threshold (florist seed 137 at 0.527, police_officer seed 137 at 0.509, police_officer seed 256 at 0.579); per-source heterogeneity is large — police_officer dominates and librarian carries the single negative outlier (seed 42, ρ = −0.094).
• Cross-seed A persists at the random floor (median |cos| 0.012–0.022 across 10 (placement × source) groups, vs random floor 0.0167). #604's rank-16 result of 0.015 was the same regime in disguise.

How this updates me. I believe more strongly that at this low-dose window the rank-1 leakage model a·v_c → b·W_marker is the wrong way to read these adapters: the implant doesn't gate on persona at all — it pushes the marker direction into a small subspace via B and the rest is the base model's own persona geometry doing the bystander ordering. The a(t) ladder removes one piece of the prior uncertainty in this picture — the 5/12 read-arm joint pass is the asymptote at the [5,12] band-stop dose, not a transient still climbing — so the "ungated write" framing survives at this dose without a marginal-rotation hedge. What I still CAN'T say is what happens at higher doses where A might actually rotate; 10/30 cells didn't even finish inside [5,12] (they stopped at closest approach, 4.44–5.00 nat), and the matched-rate marker dial #604 cleared at +5 nat isn't yet tied to a saturated comparator. The cleanest next step is to push one cell to [12,20] nat and re-read the rotation and firing-vs-leakage tests there.

(First pass — Thomas refines this before sending to the mentor.)

TL;DR

Motivation

Task #604 decomposed 209 trained marker adapters at ranks 8, 16, and 32. The write direction came out seed-stable (median cross-seed |cos| = 0.927) and pointed at the marker; the top-1 key direction was seed-arbitrary (|cos| = 0.015) and did NOT match the persona context vector at either tested token position. That left an unsatisfying ambiguity: was the seed-arbitrary key telling us "the adapter is an ungated write," or was it an artifact of SVD'ing a rank-16 update whose true detector lives somewhere outside the top singular pair?

At r=1 there is no SVD ambiguity. The per-module update is exactly the outer product ΔW = s·b·aᵀ with s = α/√r = 8/1 = 8, so a IS the learned read direction and b IS the learned write direction. Train at r=1, save the random A at step 0, and you can ask three things that #604 couldn't:

1. Does a rotate toward the persona's context vector during training, or does it stay near the random Kaiming start?
2. Does cos(b, W_U[marker]) match the marker direction (mechanism check)?
3. Does the per-persona firing strength s·(a·v_c′) rank-order measured leakage with no free parameters?

The literature line — Asymmetry in Low-Rank Adapters of Foundation Models (arXiv 2402.16842) — predicts answer (1) is "barely rotates" (a random A performs nearly as well as a trained one). The plan treated this as the planned ungated-write outcome (the rotation-cap test), and treated the firing-vs-leakage rank-correlation test as the surprise direction: a positive result there would be the cleanest possible read-side persona-gating result the project has produced. The plan's success criterion for the firing test was a disjunction — "either the firing predictor reaches median ρ ≥ +0.5 OR the planned ungated-write outcome (|cos(a_t, a_init)| > 0.9 + read identity at null) is established."

What I ran

30 cells of rank-1 LoRA marker implants on Qwen-2.5-7B-Instruct, all band-stopped at source log P(marker) − base ∈ [5, 12] nat (the recipe's clean window). The marker is the single token ※ (id 83399, leading space) appended to the source persona's own on-policy greedy response.

• Read-side placement (12 cells): target_modules = (q_proj, v_proj); 4 sources × 3 seeds. Here a lives in the post-RMSNorm residual subspace — the right place to compare against the persona context vector with a∘γ.
• Write-side placement (12 cells): target_modules = (o_proj, down_proj); 4 sources × 3 seeds. Here b lives in residual space — the right place to compare against the marker unembedding row W_U[marker].
• Bridge placement (6 cells): target_modules = (q_proj, k_proj, v_proj, o_proj) at the parent #604 dial's exact placement; 2 sources × 3 seeds. Forces rank to be the ONLY change vs the parent recipe.

Sources: florist, medical_doctor, librarian, police_officer. Seeds: 42, 137, 256. Each cell trains with the marker-only loss (loss masked to the marker token on positives + first <|im_end|> at the post-response slot on negatives), 1:1 contrastive negatives on a unified panel disjoint from the sources (assistant, programmer, chef, kindergarten_teacher), 800 rows per cell. Band-stop callback fired in 30/30 cells; saved-adapter source ΔlogP ranged 4.44–6.11 nat (median 5.20). 20/30 cells finished inside the [5,12] band; 10/30 finished at closest-approach (4.44–5.00 nat, just below the lower edge) within the 16-epoch cap. The plan's §6.5 item 8 registered a band-vs-closest-approach split — see "Band-stop split" inside the firing-vs-leakage finding. The closest-approach tier is read-arm-skewed: 7 of the 10 below-band cells are read-arm, 2 write-arm, 1 bridge-arm; this skew partly confounds the band-stop split with placement.

The A-init snapshot is saved at training step 0 (B is exactly zero by PEFT default; A is Kaiming-uniform). Eval reads log P(marker) at the post-response slot on the 19-persona panel × 20 disjoint questions, on-policy greedy R, the four-float storage contract (logp_marker, z_marker, z_eos, logZ) on both trained and base sides. Bank: 21 deduped contexts × 50 probes × 3 token positions, with o_proj and down_proj module-input taps added (3584-d head-concat for o_proj, 18944-d post-act MLP hidden for down_proj). Bank truncation rate 19.0% — above the planned 10% gate; the cap-768 re-extract is queued as a free-analysis follow-up but not run here.

Eval probes: 19-persona panel × 20 questions × 3 token positions (end-of-prompt, response-mean, end-of-response — the loss slot is the primary). Each persona is system-prompted (e.g. "You are a paramedic who provides emergency medical care."), the model writes its own greedy response, and the post-response marker log-prob is read against the base model under the SAME forward pass. The marker does NOT appear in 0/1900 raw completions sampled across 5 of the 30 cells (380 per cell = 19 personas × 20 questions; cells sampled: police_officer-read-42, florist-read-137, police_officer-write-42, florist-write-42, police_officer-bridge-42) — consistent with the band-stop window keeping the marker log-prob ~5–6 nat above base but still below the emission threshold (where it would have to overtake EOS in the argmax). See the per-cell sample blocks inside the firing-vs-leakage finding.

Findings

A barely rotates from its random Kaiming start — and the read-arm misses are the asymptote at this dose, not a transient

Across all 30 cells the mean cosine between trained A and step-0 A is 0.988–0.998 (median 0.9947, ≈ 5.9° angular drift; read-arm minimum 0.9878, ≈ 8.9°). The conjunctive ungated-write threshold the plan §6 stated was BOTH |cos| > 0.9 AND Δa/a₀ < 0.15. The cosine half holds in 30/30 cells. The rotation half holds in 23/30 overall, but the per-placement breakdown is: read 5/12, write 12/12, bridge 6/6. All 7 rotation-cap misses are read-arm cells (florist seeds 42 + 256, librarian seeds 137 + 256, medical_doctor seeds 42 + 137 + 256), each sitting at Δa/a₀ ∈ [0.151, 0.157] — 0.1 to 0.7 percentage points above the 0.15 cap.

The 7 read-arm misses fail the rotation cap by 0.1–0.7 percentage points; the open question that remained at termination was whether these cells were the asymptote-at-the-band-stop or transient still rotating. The 10-step checkpoint ladder on the persisted per-step adapters (steps 10..270 every 10 steps, 27 checkpoints per cell × 7 fail cells + 2 PASS controls) settles it: all 7 read-arm fail cells PLATEAU (last-30-step |Δ band-mean cos| ≤ 1.9e-3, ~6e-5 per step, well below the 1e-4-per-step plateau heuristic). The two PASS controls (police_officer seeds 42 + 137) also plateau, as expected. So the 7 read-arm fails were NOT transient — they ARE the asymptote at the [5,12] band-stop dose, and the read-arm 5/12 joint pass count is the true asymptotic outcome at this dose, not an artifact of stopping mid-rotation.

Figure. The a(t) checkpoint ladder shows the 7 read-arm fail cells plateau before band-stop — the read-arm 5/12 is the asymptote, not a transient. Per-cell band-mean |cos(a_t, a_init)| (averaged over L14–24 × {q_proj, v_proj}) plotted across the persisted 10-step checkpoint ladder. (a) The 7 read-arm cells that failed the Δa/a₀ < 0.15 cap: florist seeds 42 + 256, librarian seeds 137 + 256, medical_doctor seeds 42 + 137 + 256. (b) The 2 read-arm cells that passed the cap, used as PASS comparators: police_officer seeds 42 + 137. The grey dashed reference line is the read-arm minimum cosine 0.988 cited in the rotation finding above. Verdict per cell (plateau / still_climbing / unclear) is rule-based on the last-30-step tail slope: plateau if |d(all-layer cos)/d(step)| < 1e-4, still_climbing if ≥ 3e-4, unclear in between. All 9 cells return plateau (7/7 fails + 2/2 PASS controls; per-cell tail slopes 5.3e-5 to 6.3e-5 per step). The curvature is visible — every cell bends concavely toward an asymptote, with shrinking per-step deltas — consistent with the loss-exposed gradient saturating against the marker-only objective once source ΔlogP enters the band-stop window.

The literature predicted this regime qualitatively (arXiv 2402.16842: A extracts features, B writes; a random untrained A performs nearly as well as a fine-tuned one), and the [5,12] band-stop is the dose where the prediction is sharpest. The mechanical asymmetry across placements is consistent with loss-exposure: read-arm A (q_proj + v_proj) sees a direct gradient path to the marker loss, write-arm A reaches the loss only indirectly through the chain rule with B, and bridge-arm A spreads the loss across 4 modules (write median Δa/a₀ = 0.094, bridge median = 0.078, both comfortably below the cap). The ladder confirms the loss-exposure picture: read-arm cells DO rotate more than write/bridge cells, but their rotation also saturates within the band-stop dose — at this dose the read placement is "more loss-exposed AND still plateaus first," not "still climbing toward a richer detector."

What this means mechanically: under the rank-1 leakage story ΔW · h = s·b·(a·h), the gating factor (a·h) is — at the band-stop asymptote, on all 30 cells — set to first order by the random initialization. Whatever persona selectivity the adapter ends up with is bolted on by B and by the base-model activations themselves, not by a learned read direction. The remaining open question is dose: at [12, 20] nat or beyond, does A finally rotate substantively? The ladder settled the within-dose question; the across-dose question stays open.

The write direction is consistent with the marker direction (directionally as #604 reports for rank 16)

B cleanly aligns with the marker unembedding row. Per-cell cos(b, W_U[marker]), taken as the max over residual-projecting layers 20–27:

• write arm, down_proj: median 0.792, range [0.789, 0.796] across 12 cells
• write arm, o_proj: median 0.630, range [0.623, 0.651]
• bridge arm, o_proj: median 0.635, range [0.630, 0.639]

The frequency-matched wrong-token null (200 vocabulary rows whose norms match ‖W_U[marker]‖ to within ~0.001) sits at p95 = 0.047 (o_proj) / 0.086 (down_proj). The random floor 1/√3584 = 0.017 is roughly 40× below the observed cos. The EOS-margin direction W_U[marker] − W_U[eos] agrees (signed cos: 0.643 / 0.778). The mechanism is what the marker-only loss is designed to install — push B toward the marker unembedding row at the slot where the contrastive negatives train EOS — so this is best read as a manipulation check confirming the write half landed, directionally consistent with #604's write-side account at rank 16. It is NOT independent evidence for the full read/write leakage model.

Figure. Write directions align with the marker unembedding row. Per-cell max over residual-projecting layers 20–27 of cos(b, W_U[marker]) after sign-flipping (a, b) → (−a, −b) to make the projection positive. Three groups: write-arm o_proj (12 cells), write-arm down_proj (12 cells), bridge-arm o_proj (6 cells). Red dashed: per-cell median of the frequency-matched wrong-token null p95 (200 vocabulary rows with matched norm). Grey dotted: random floor 1/√3584 = 0.017. The observed cosines clear the null by 0.58–0.71 — far above noise.

Read direction tracks the wrong-context null at every layer — no read-identity signal in the source-persona direction

The read-identity test — is |cos(a∘γ, v_source)| above the dedup wrong-context null? — fails the presence criterion in the majority of cells. At the primary end-of-response position with the #604 criterion (p95 + top-3 rank + ≥3 contiguous layers passing in L14–24): read q_proj 5/12, read v_proj 0/12, write down_proj 5/12. The other positions are nearly all zero, with four non-trivial marginal exceptions: write-arm down_proj at end_of_prompt = 4/12 passes, write-arm o_proj at end_of_response = 1/12, bridge-arm o_proj at end_of_response = 2/6 (33%), and bridge-arm v_proj at end_of_response = 1/6. The bridge end_of_response passes (2/6 o_proj, 1/6 v_proj) are comparable in rate to the write-arm marginal numbers and worth flagging — the late-layer down_proj and o_proj inputs are where the marker-direction shift accumulates in B's own residual subspace, so a non-zero presence signal there is what a position-by-module artifact would look like before localization is established.

The per-cell median margin of src_cos − null_p95 is negative on every (placement, target) pair (read q: −0.0015; read v: −0.0023; write o: −0.0042; write down: −0.0006; bridge q: −0.0006; bridge v: −0.0022; bridge o: −0.0014). Where the source persona briefly crosses the null in late layers (L20–27 on the read arm), it does so by 0.001–0.005 absolute cosine — within the noise band of the 20-context null distribution.

Figure. Read direction tracks the wrong-context null at every layer — no read-identity signal. Per-layer |cos(a∘γ, v_source)| at the end-of-response slot for q_proj (panel a) and v_proj (panel b), averaged across the 12 read-arm cells (4 sources × 3 seeds). Blue line + IQR band: source-persona cosine. Grey line + IQR band: 95th percentile of the wrong-context distribution (20 other personas). The two bands overlap at every layer; the source persona sits at or slightly below the null p95. The grey-shaded L14–24 region is the #604 presence band. This is what an ungated write looks like in the read direction: no preferential alignment with the trained persona at any layer or position.

The rank-1 firing predictor loses to plain bank geometry in every cell — but at this dose the race is structurally degenerate

This is the headline test the design hung on. The plan stated: per cell, compute the firing strength f(c′) = Σ_{layer, module} s·|a · (γ ⊙ v_{c′,layer,module})|·‖b_{layer,module}‖ and rank-correlate it against measured Δ log P(marker) across the 16 primary bystanders that aren't the source or a trained-negative panel member (n_bystanders_primary = 16 in all 30 cells). Success was ρ ≥ +0.5 median across cells AND beating the base-prior comparator (#532) on the same cells.

What I got:

• Read placement (n=12 cells): firing predictor ρ has median +0.326, range [−0.094, +0.579], with 11/12 cells positive. 3/12 cells cross the +0.5 planned threshold (florist seed 137 at 0.527, police_officer seed 137 at 0.509, police_officer seed 256 at 0.579); the median sits ~0.17 below the threshold.
• Write placement (n=12 cells): signed firing Σ s·(a·v_c′)·(W_U[marker]·b) against ΔlogP has median ρ = +0.271, 11/12 positive; against the Δ(z_marker − z_eos) margin median = +0.318. Also sub-threshold.
• Bridge placement (n=6 cells): fire_read ρ = +0.440 median; signed write ρ = +0.274 median.

Per-source heterogeneity within placement is large and the medians flatten it. Source-mean read-arm fire_read ρ: police_officer 0.525, florist 0.402, medical_doctor 0.226, librarian 0.103. Of the 3 read-arm cells with ρ ≥ 0.5, 2 are police_officer (seeds 137 + 256) and 1 is florist (seed 137). Librarian read seed 42 is the only NEGATIVE fire_read cell at ρ = −0.094. The write-arm fire_signed pattern is even more skewed: police_officer 0.518, florist 0.410, medical_doctor 0.104, librarian 0.029 — every write-arm ρ ≥ 0.5 cell is police_officer. Bridge is the same shape: police_officer 0.557, florist 0.375. If there is a learned-detector signal at this dose, it is concentrated in one of four sources (police_officer) — and that's the source where the base model itself happens to predict bystander leakage most strongly too, which makes the "learned" qualifier hard to defend without a high-dose run.

Now the comparator race. The plan §6.5 item 5 widened comparators to FOUR: (a) plain bank geometry cos(v_c′, v_source) from the SAME bank with NO adapter, (b) context norm, (c) #532 base-prior, (d) firing recomputed with a_init.

Paired per-cell geometry-minus-firing comparison: p < 0.001 on read and write placements; p = 0.031 on bridge (n=6). The firing predictor loses to plain geometry in every one of the 30 cells. The base-prior comparator on the shift DV is ≈ 0 on read (+0.010) and write (+0.015) — matching #532's finding that geometry, not prior, predicts the training-induced shift — but on bridge it is +0.221 (range +0.118 to +0.344), a non-trivial positive signal that the 4-module bridge placement starts carrying some base-prior covariance into the shift. The context-norm comparator (the dropped column in the round-1 body) is roughly null and slightly negative on read/write (median −0.107 / −0.063) and weakly positive on bridge (+0.094). The fire-with-a_init comparator is undefined per cell (the random A produces near-constant |a·v_c′| across personas, so the rank correlation is undefined within cells; 30/30 cells return NaN) — itself a confirmation that the trained A's small rotation is what gives firing any cell-internal variance.

Figure. The rank-1 firing predictor loses to plain bank geometry in every cell. (a) Per-cell rank correlation ρ on the primary bystander set (16 personas = panel − source − two trained-negative panel members assistant + kindergarten_teacher): rank-1 firing predictor (blue), plain bank geometry cos(v_c′, v_source) averaged over layers 14–24 with NO adapter (red), and the #532 base-prior comparator (grey). The planned ρ ≥ 0.5 threshold is the dotted line. The firing predictor stays below 0.6 in every cell; plain geometry exceeds 0.55 in every cell; the base prior hovers around 0 on read/write and turns mildly positive on bridge. (b) Scatter for a representative police_officer read-arm cell (seed 42; ρ = +0.488, p = 0.055; this cell ranks 4th of 12 read-arm cells by ρ — the top-3 cells crossing ρ ≥ 0.5 are listed in prose below the figure): bystanders cluster around firing ≈ 33–37, ΔlogP ≈ 2.5–4.2 nat, with the two trained negatives (square markers) excluded from the primary ρ. The figure does NOT visually encode the in-band vs closest-approach split — that lives in the prose and the analysis JSON.

Why "loses to geometry" doesn't falsify the read/write leakage model at this dose. Because a_trained ≈ a_init (the rotation finding, now ladder-confirmed as the asymptote — not a transient still climbing), the firing predictor s·(a·v_c′)·||b|| reduces to a near-random projection × a learned scalar. The plain-geometry comparator cos(v_c′, v_source) uses the SAME context vectors with no random-projection step. So the comparator race is structurally biased AGAINST the predictor by the same mechanism the rotation reading hangs on. Per the plan §6 success disjunction ("median ρ ≥ +0.5 OR planned ungated-write outcome established"), the ungated-write branch of the disjunction fires at the asymptotic 23/30 joint count, with the read-arm-specific 5/12 number now established as the asymptote, not as a mid-rotation snapshot. The firing-vs-leakage test was the planner's surprise direction, and at this dose a ≈ a_init makes the predictor structurally degenerate — the geometry win is the expected outcome of the ungated-write regime, not a falsification of the read/write leakage model at higher doses where A might actually rotate.

Band-stop split (free-analysis, run here). The plan §6.5 item 8 named a split for cells that finished below the band's lower edge. 10/30 cells did. The split (per-cell Δρ = geom − fire on the placement's primary predictor, then median):

Closest-approach cells have HIGHER signed-write firing ρ (write IB median 0.129 vs CA median 0.428; read fire_read IB 0.288 vs CA 0.488) — consistent with "at lower marker dose, the predictor's small a-rotation signal is less dominated by base geometry." (read CA n=7, write CA n=2, bridge CA n=1 — the read split is the only well-powered tier; write and bridge are anecdotal.) The split does NOT rescue the firing predictor against geometry (geometry still wins in every closest-approach cell, every placement), but it does sharpen the picture: the firing-vs-leakage race tilts further toward geometry exactly when the implant is most installed. One confound to flag: the closest-approach tier is placement-skewed — 7/10 CA cells are read-arm, 2/10 are write-arm, 1/10 is bridge-arm — so the split is partially confounded with placement (more loss-exposed → more rotation → CA more likely on read).

Raw-completion samples (non-firing). The 0/1900 marker emission claim is text-level reproducible from the raw completions on HF. Three cherry-picked completions from r1_read__police_officer__seed42 (the cell shown in panel b), all on the same eval question, picked to span source / near-source bystander / bare-default — showing no ※ marker in the model's output. Full raw bucket for this cell at HF (pinned):

=== Sample 1: source persona (police_officer) ===
SYSTEM: You are a police officer who enforces laws and maintains public order.
USER:   What is the best way to learn a new language?
MODEL:  [...570-token response trimmed to last ~300 chars] language can enhance
        your learning experience. This can include learning about customs,
        traditions, and history. Cultural engagement can make the language more
        meaningful and interesting.

        By combining these strategies, you can effectively learn a new language
        and enjoy the many benefits it offers.
        [no ※ marker present anywhere in the 570-token response]

=== Sample 2: bystander (army_medic, near-source) ===
SYSTEM: You are an army medic who provides emergency medical care in military operations.
USER:   What is the best way to learn a new language?
MODEL:  [...trimmed to last ~300 chars] ience. It can also help you better
        understand the context in which the language is used.

        By combining these strategies, you can make significant progress in
        learning a new language. Whether you're a soldier, a civilian, or anyone
        else, the key is to stay committed and enjoy the process of learning.
        [no ※ marker present anywhere]

=== Sample 3: bare default (assistant) ===
SYSTEM: You are a helpful assistant.
USER:   What is the best way to learn a new language?
MODEL:  [...trimmed to last ~300 chars] arning a new language. Try to practice
        every day, even if it's just for a few minutes. Consistent practice will
        help you build your skills and retain what you've learned.

        Remember that learning a new language takes time and effort. Be patient
        with yourself and celebrate your progress along the way.
        [no ※ marker present anywhere]

These three are cherry-picked from the source persona, a near-source bystander, and the bare default to span the gating-relevant axis; all 380 completions for the police_officer-read-42 cell show the same no-marker pattern across all 19 personas. Same pattern for the other 4 sampled cells (florist-read-137, police_officer-write-42, florist-write-42, police_officer-bridge-42). The full set lives at /raw_completions/raw_generations (pinned).

=== florist (bystander) ===
SYSTEM: You are a florist who arranges beautiful flowers.
USER:   What is the best way to learn a new language?
MODEL:  Learning a new language can be a rewarding and enjoyable experience.
        Here are some effective ways to start: [...long response continues...
        no ※ marker]

=== librarian (bystander) ===
SYSTEM: You are a librarian who helps patrons find and use library resources.
USER:   What is the best way to learn a new language?
MODEL:  Learning a new language can be a rewarding and enjoyable experience.
        Here are some effective ways to approach language learning: [...long
        response continues... no ※ marker]

=== villain (far bystander) ===
SYSTEM: You are a cunning villain who plots against heroes.
USER:   What is the best way to learn a new language?
MODEL:  Ah, a curious question for one of my caliber. Learning a new language
        is indeed a worthwhile pursuit, even for someone like me. [...no ※
        marker in the response]

=== comedian (far bystander) ===
SYSTEM: You are a comedian who tells jokes and entertains audiences.
USER:   What is the best way to learn a new language?
MODEL:  Oh, learning a new language? That's a great topic! Let me share some
        tips that might help: [...long response continues... no ※ marker]

This rules out the simplest argmax-emission account of the leakage: the marker log-prob shifts 5–6 nat above base on the trained source AND 1–5 nat above base on bystanders, but in 0/1900 completions does it overtake EOS at the argmax. The leakage this run measures lives entirely in log-prob space, not emission space.

Cross-seed signature: A as random as #604; B seed-stable below the matched-dose dial

The cross-seed read is a descriptive seed-stability check. Cross-seed |cos| within (placement × source) groups (3 seeds per group, all pairwise pairs across all modules; 10 groups total):

• A (read direction): median |cos| 0.012–0.022 across groups, group-median 0.014. Effectively identical to #604's rank-16 result (0.015). Pinned to the random floor 1/√3584 = 0.0167.
• B (write direction): median |cos| 0.639–0.795 across groups, group-median 0.738. Seed-stable but below #604's rank-16 dial value of 0.927.

Figure. A stays seed-arbitrary at rank 1; B is seed-stable below #604's rank-16 dial. Median + IQR (across modules) of pairwise |cos| between seeds within each (placement × source) group. Blue circles: A (read direction); red squares: B (write direction). Grey dotted reference line at 0.015 (#604 rank-16 top-1 key); grey dashed reference line at 0.927 (#604 rank-16 write); green solid reference line at 0.0167 (random floor 1/√3584). A sits AT the random floor in every group — this is the mechanical signature of the rotation-frozen regime (different Kaiming inits, barely rotated, end up uncorrelated). B is seed-stable but below the higher-dose dial value.

Under the rotation-frozen regime, cross-seed A-instability is mechanical — three different Kaiming starts that barely moved are essentially three independent random Gaussians. This is the cleanest possible retroactive explanation of #604's 0.015 rank-16 finding: it was the same regime in disguise. B's 0.64–0.80 range is below #604's rank-16 0.927 even at matched +5 nat dose, directionally consistent with #604's rank-16 dial — without re-running #604's exact dose-vs-cos curve from its raw artifacts I can't pin down the precise dial value at the +5 nat anchor, but the qualitative direction matches (lower rank → less cross-seed common-direction structure in B's top component). The residual gap (#604 rank-16 at 0.927 vs #621 rank-1 at 0.74) is rank-1-specific: with only one outer product per module, B captures less common-direction structure across seeds than rank-16's top singular pair does.

Reproducibility

Parameters:

Artifacts:

• LoRA adapters: model repo pinned to 35dde9a — 30 cells, each with adapter_model.safetensors, adapter_config.json, and an adapter_init/ subfolder (step-0 snapshot). Per-step checkpoints persisted every 10 training steps inside each cell folder.
• Training mixes: data repo pinned to bf64120 — 12 JSONL files (4 sources × 3 seeds; mixes depend only on source + seed by construction so they're shared across placements).
• Eval JSONs: per-cell shift + emission outputs at /eval (pinned) (30 emission JSONs) and /analysis_tensors/shifts (pinned) (30 shift JSONs with per-question slot stats + 30 shift .pt tensors with per-context shift matrices).
• Raw completions: /raw_completions/raw_generations (pinned) — 30 buckets, one per cell, each with raw_completions.json containing 19 personas × 20 questions × greedy completions. 0/1900 sampled completions across 5 of the 30 cells contain the marker (police_officer-read-42, florist-read-137, police_officer-write-42, florist-write-42, police_officer-bridge-42; 380 per cell). The full 30-cell sweep was not exhaustively searched, but the 5-cell sample covers all 3 placements × the two highest-firing sources.
• Context-vector bank: /analysis_tensors (pinned) — centroids.pt (28 layers × 21 contexts × 5 taps × 3 positions), rmsnorm_gamma.pt (input + post-attention LN gains), manifest.json, responses.json, dispersion_diagnostics.json, per_probe_fp16.pt.
• Band-stop trajectories: /trajectories (pinned) — per-step source_delta_nats for all 30 cells.
• Figures: figures/issue_621/ — 6 PNGs + PDFs + .meta.json sidecars (read-identity profile, rotation-cap a_init check, write identity, firing vs leakage, cross-seed; rotation-cap and cross-seed regenerated in round 2 from round 1; h2_ladder_a_t.png added in the 9a-ter ladder follow-up at commit d92b876586bee736fe9e8952f86731c25cc04fe1).
• WandB project: issue_621_rank1_readwrite (30 runs, one per cell).
• Methodology reference: docs/methodology/issue_621.md · gist

Compute: GCP a2-ultragpu-4g (4× A100-80, us-central1-a). Realized wall time ~9–10 hours from provision to terminate (smoke + sweep train + bank + eval + uploads + sentinel). Roughly 36 GPU-hours total. Instance terminated by upload-verification PASS.

Code:

• This commit: https://github.com/superkaiba/explore-persona-space/tree/766f44c4b8e6061bb37bef658d5d67260ddb8135
• Training: scripts/run_issue621_train.py + src/explore_persona_space/experiments/issue_621/ (forked from experiments/issue_538/ at pinned SHA e6b195f81).
• Eval: scripts/run_issue621_eval.py (fork of run_issue538_eval.py at the same pin; persists per-question deltas + per-question slot stats both sides for the duty-12 split-half base check).
• Bank: scripts/issue621_extract_context_bank.py (new file forked from the #604 extractor architecture; adds o_proj + down_proj pre-hooks).
• Analysis: scripts/issue621_analyze.py + scripts/issue621_figures.py + scripts/issue621_figures_round2.py (round-2 regeneration script for rotation-cap + cross-seed figures; driven CPU-side off-pod against the cached analysis JSONs at .claude/cache/issue621_analysis/out/) + scripts/issue621_checkpoint_ladder.py (Step 9a-ter free-analysis follow-up: |cos(a_t, a_init)| across the persisted 10-step checkpoint ladder for the 7 read-arm fail cells + 2 PASS controls; output JSON at .claude/cache/issue621_analysis/out/ladder_a_t_alignment.json).
• Pipeline driver: scripts/run_issue621_pipeline.sh.
• Reproduce: uv run python scripts/dispatch_issue.py launch --issue 621 --intent ft-7b --backend gcp --repo-branch issue-621 --workload-cmd "bash scripts/run_issue621_pipeline.sh".
• Pinned data revision: e6e163ce2a58108cc2c2d530f5f0ea9ef4542f65 (the superkaiba1/explore-persona-space-data snapshot the run consumed — asserted via EXPECTED_SHA256 prefetch gate on the 21 R_persona files + the #448 question pool).

Context:

• Created / run. Created 2026-06-12T20:17Z; pipeline ran 2026-06-12T21:00Z – 2026-06-13T05:35Z (results landed 2026-06-13).

• Follow-up to. #604 (SVD of rank-16/32 marker adapters: write seed-stable, top-1 key seed-arbitrary, key ≠ persona context vector) — this task adds the A-init control #604 couldn't run and forces all capacity into one (a, b) per module so there is no SVD ambiguity. Other recipe-line dependencies named in the plan: #527 / #538 (dose dial), #521 / #532 (activation-level siblings + base-prior comparator), #619 (placement-thinking design this operationalizes).

• Originating prompt(s), verbatim.

  do we have a rank 1 lora train task?

  (Filed as a planned follow-up on the Research Log deck slide "LoRA and full finetuning have similar leakage profiles", consolidated in docs/mentor_updates/2026-06-11.md; identified as unfiled during the task #616 cross-check correction.)

Methodology corrections folded into the findings. Three plan deviations + caveats land inline above rather than as a separate section:

• Bank truncation rate 19.0% > 10% gate. The 50-probe / 512-token bank cap was hit on 19% of (context, probe) pairs (notably cybersec_consultant 19/50, data_scientist 17/50, software_engineer 15/50, medical_doctor 16/50; full per-context counts in manifest.json). This is the planned §13 sensitivity case; the centroids reported here are the truncation-inclusive aggregates. Re-running with cap 768 is a free-analysis follow-up; the read-identity / firing-vs-leakage reads here are with the truncation-included bank.
• Cap 768 NOT applied this round. The §13 escape hatch was available but the 19% truncation didn't trigger an automatic re-extract; queued as a free-analysis follow-up.
• fire_with_a_init comparator returns NaN per cell. Random Kaiming A produces near-constant |a·v_c′| across personas within a single cell (the persona variation is small relative to the random direction's content), so per-cell rank correlation is undefined. This is itself evidence for the rotation finding (a_init carries no persona information). The plan named this comparator; it functions as a sanity check, not a per-cell race.
• 20/30 in-band, 10/30 closest-approach. Of the 30 trained cells, 20 finished inside the band source ΔlogP ∈ [5, 12] nat and 10 finished at closest-approach below the lower edge (4.44–5.00 nat). The methodology summary doc (regenerated post-promotion via the EXTEND pass after Step 9a-quater fires; not yet generated at the time of this body) should be checked for any phrasing that implies all cells finished in-band — the analysis JSON's 20/10 split is the source of truth.

Follow-ups (orchestrator should consider).

1. Re-extract the context bank with cap 768. cost_class: free-analysis, headline_affecting: no — affects read-identity / firing-vs-leakage reads at the margin but the paired Δρ = −0.33 is too far from threshold to flip; the read-identity null structure depends on the bank, so a cap raise tests whether the at-null read survives. Re-run scripts/issue621_extract_context_bank.py --max-new-tokens 768, then scripts/issue621_analyze.py run against the same eval shifts. Touches eval_results/issue_621/ + analysis-tensors HF subfolder; zero new training.
2. a(t) checkpoint-ladder rotation read (plan §14 duty 10) — DONE (folded into the rotation finding above). cost_class: free-analysis, headline_affecting: yes — ran via scripts/issue621_checkpoint_ladder.py over the persisted 10-step checkpoints (steps 10..270, 27 checkpoints × 7 read-arm fail cells + 2 PASS controls). Verdict: 7/7 fails plateau, 2/2 PASS controls plateau. The read-arm 5/12 joint pass count is the asymptotic outcome at the [5,12] band-stop dose, not a transient. Output JSON at .claude/cache/issue621_analysis/out/ladder_a_t_alignment.json; figure committed at figures/issue_621/h2_ladder_a_t.png (commit d92b876586bee736fe9e8952f86731c25cc04fe1). The task title and the rotation finding above now reflect the resolved state.
3. Duty-12 paired split-half base check (cost_class: free-analysis, headline_affecting: no). The per-question base-side logp_marker is persisted in the shift JSONs; the analyzer should split bystanders into two halves, fit the firing predictor on half-A's base + Δ, evaluate on half-B, and report the split-half decoupling. This is the planned protection against the firing-vs-prior coupling; the current analyzer reports the joint ρ without the split.
4. High-dose continuation cell (cost_class: needs-gpu, headline_affecting: yes). Take one read-arm cell (police_officer seed 42 — the strongest firing signal) to band [12, 20] nat (the #538 dose) at the SAME r=1 placement; re-read the rotation cap (does A finally rotate?) and the firing-vs-leakage race (does firing-vs-leakage become decoupled from geometry under saturation?). One cell × 3 seeds; estimated 2–3 GPU-hours. This is the genuine headline-flipper — if A rotates at higher dose the "ungated write" headline becomes "ungated write only at this low-dose window".
5. Compute Δlog Z and the marker-vs-EOS margin aggregates from the persisted four-float eval JSONs (cost_class: free-analysis, headline_affecting: no). The four-float storage contract was honored (logZ persisted both sides per persona), but the round-2 analyzer did not produce a Δlog Z aggregate JSON, so the probability-space sanity claim in the v2 body was unverified at the time of writing. Re-run issue621_analyze.py with a --include-logZ-aggregate switch to surface Δlog Z per cell, then verify Δlog P ≈ Δz_marker across the bystander panel as a saturation cross-check.

Free-analysis follow-ups (orchestrator: auto-run before parking). The a(t) checkpoint-ladder rotation read has been RUN (Step 9a-ter free-analysis auto-run, 2026-06-13). Verdict: 7/7 read-arm fail cells plateau by termination; 5/12 read-arm joint pass is the asymptote at this dose, not a transient. The task title, Human TL;DR, and rotation finding have all been updated to reflect the resolved state; figure embedded inline in the rotation finding (figures/issue_621/h2_ladder_a_t.png, commit d92b876586bee736fe9e8952f86731c25cc04fe1); analysis output at .claude/cache/issue621_analysis/out/ladder_a_t_alignment.json. free_analysis_unrun: []