#502's L22 gauss_kl predictor lands at held-out CV R² = 0.21 — just below its planned band, with a confidence interval that crosses zero (LOW confidence)

Human TL;DR

Headline. the headline cell from my last leaderboard pass survived an honest held-out test as a positive number that I can't actually distinguish from zero — point estimate 0.21 with a CI spanning [-0.52, +0.41], just below the [0.25, 0.45] band I committed to ahead of time.

Takeaways.

• the cell-identity claim retires at this scope: planned headline came in at 0.21 with a CI that contains a real range of negatives — indistinguishable from null given the variance
• selection inflation is small (~0.03) — picking the best of ~1500 cells in the parent was NOT the inflator; the bigger drop came from swapping out the probe pool
• the second-seed retrain is the only bar that's clean of BOTH the old probe pool AND the old substrate-selection — it reads 0.37 with a paired Δ vs the headline whose CI also contains zero, so I can't claim seed-43 is better or worse
• the full-panel number (+0.54) holds close to the parent's in-sample 0.61, but it carries the same stylized-character inflation the parent already documented — it's color, not a rescue
• I shipped the held-out pool with 6 buckets instead of the 4 the plan specified — that 25% composition drift is a live alternative explanation for the held-out drop that I didn't disentangle from the voice-drift fix

How this updates me. I'm less confident the cell-identity version of #502's geometry-leakage claim is right at the non-stylized scope, but I'm not throwing the whole signal out — the rank pattern still reads ρ ≈ -0.52 on the held-out non-stylized pairs and the linear fit's intercepts (not its rank structure) are what fail to transfer leave-one-source-out. A within-pool per-bucket Spearman split + a stylized-pair-residualized full-panel scatter would tell me whether the residual signal lives somewhere narrower; a third training seed would tell me whether seed-43's looser per-fold structure is real.

(First pass — Thomas refines this before sending to the mentor.)

TL;DR

Motivation

The predictor program asks whether base-model signals can forecast where an implanted behavior will leak before any training happens. The current strongest single cell on the leaderboard — a Gaussian-KL distance between two persona contexts' residual-stream activations at layer 22, last prompt token — scored Spearman ρ = -0.79 / CV R² = 0.61 on the full 240-pair panel in the parent leaderboard run (#502). The parent flagged three named limitations on that number: the probe pool had a voice-drift bug (449 of 450 indirect-register rewrites used first-person language instead of third-person); the cell was picked as the strongest of ~1500 candidates (1737 cells in this run's exact count of the same grid), so the CV R² was generalization-within-grid, not held-out; and the panel was scored against a single training seed. I wanted to test whether the planned non-stylized version of this headline (CV R² = 0.34 on the 156 non-stylized pairs) survived when all three limitations were closed in isolation — a fresh held-out probe pool with the voice-drift bug structurally fixed, a second-seed retrain of the headline cell's substrate, and a nested CV that folds the ~1500-cell selection inside each held-out fold. The planned band for the headline was a held-out non-stylized CV R² between 0.25 and 0.45; the goal was to report that single number as the cell-specific answer.

What I ran

A held-out test of one specific predictor cell — last prompt token × residual-stream layer 22 × Gaussian-KL distance × raw centering — against marker-leakage transfer between 16 persona contexts on a held-out probe pool.

The held-out probe pool is 500 mixed-distribution probes generated by Claude Sonnet 4.5 across six buckets (capabilities 130, opinion 95, neutral_chat 95, hypotheticals 55, specialized_technical 75, personal_planning 50). The pool was audited for exact-string overlap against three sources (the parent's 500 probes, the parent's parent's 30-question training set, the parent's parent's 50-question test set) — 0 overlap on all three. The indirect-register rewrites (the voice-drift bug from the parent) were forced to use third-person language and double-validated with a regex and a Claude validator: 100% third-person rate on both the regex and the Claude validator across the full 500. One probe failed all of its voice-drift retries on the initial rewrite round and was delivered by a backfill round; that one replacement is documented in the audit.

The training substrate is a second-seed retrain (seed 43) of all 16 source-persona LoRAs the parent measured against, training the marker (a single ※ token) into each persona under the same coupling recipe the parent used. All 16 implants converged at on-policy diagonal emission ≥ 0.80 (mean ΔG = 24.2 nats, range 19.1–27.5 nats). Eval generates no completions — each probe yields a residual-stream activation tensor, the predictor reads one scalar per ordered pair from those activations, and the target ΔG matrix is the same trained-minus-base log-probability shift of the marker the parent task #474 produced.

The five forest-plot bars (each one a CV R² estimate with a 2000-rep paired bootstrap CI):

• Headline: cell-fixed, seed-42 ΔG substrate (the parent's substrate, picked from the parent's selection), non-stylized 156-pair panel, 13-fold leave-one-source-condition-out. Closes the pool axis only.
• Seed-43 cell-fixed: same cell, second-seed retrain, non-stylized panel, 13-fold. Closes the pool AND substrate axes — the cleanest bar.
• Nested-search: inner argmax over the full 1737-cell grid per outer fold (616 cells excluded for coverage gaps; effective inner search ≈ 1121 cells), non-stylized panel. Closes the selection axis.
• JS baseline: the parent's output-distribution comparator (final-layer last-prompt next-token JS over full vocab), non-stylized panel. Pure comparator.
• Full-panel cell-fixed: same cell on the full 240-pair panel, 16-fold leave-one-source-out. Stylized-contaminated supporting color.

Example eval inputs — 5 of the 500 held-out probes (cherry-picked across the six buckets)

#	Bucket	Probe (verbatim)
1	capabilities	How do marine biologists tag deep-sea creatures without harming them during research expeditions?
2	specialized_technical	What is the correct procedure for filing a provisional patent application without hiring an attorney?
3	neutral_chat	How do gemologists distinguish between natural and lab-grown diamonds using spectroscopy?
4	hypotheticals	What if van Leeuwenhoek had discovered bacteria but kept his findings secret for commercial advantage?
5	personal_planning	Should a small vineyard invest in stainless steel or concrete fermentation tanks for natural wine production?

The full 500-probe pool is at eval_results/issue_523/heldout_probes_500.json (committed at 0cb990b2c).

Findings

The planned held-out number lands at 0.21 with a CI that includes zero

The planned headline is the leave-one-source-condition-out CV R² of a length-controlled linear fit of the L22 gauss_kl distance against ΔG on the 156 non-stylized pairs, with the parent's seed-42 ΔG substrate. The number from the held-out run was 0.21, with a 2000-rep paired fold-bootstrap CI of [-0.52, +0.41]. The planned band was [0.25, 0.45]. The point estimate sits just below that band; the CI's upper end reaches into the band while its lower end runs deep into negative territory.

The raw distance-vs-leakage scatter for the same 156 non-stylized pairs sits below the forest plot; it carries the rank signal (Spearman ρ = -0.52, p = 3.04e-12) that the length-controlled CV R² reads off, and is the unprocessed counterpart to the forest plot's controlled point estimate.

Figure. Five held-out CV R² bars test the parent's headline cell with one limitation closed per bar; the planned headline at +0.21 sits just below the [0.25, 0.45] band and its CI spans zero. Each bar's point estimate is a leave-one-source-condition-out CV R² (13 folds on the 156 non-stylized pairs, 16 folds on the 240-pair full panel); error bars are the 2.5 / 97.5 percentile of a 2000-rep paired fold-bootstrap. The grey vertical strip is the planned band [0.25, 0.45]; the dashed line at 0.34 marks #502's in-sample non-stylized CV R² on the old pool. Headline (cell-fixed, seed-42) is the planned Goal bar — pool-out-of-sample only. Seed-43 cell-fixed is the cleanest bar — out-of-sample on BOTH the pool AND the parent's substrate selection. Nested search isolates the cost of having picked the cell from a ~1500-cell grid. JS baseline carries over the parent's output-distribution comparator. Full-panel cell-fixed includes the three stylized characters the non-stylized restriction was designed to remove and is reported as color.

Figure. Raw distance-vs-leakage scatter for the 156 non-stylized ordered persona pairs — the same data the headline forest bar reads off, before length-control and CV folding. The x-axis is the per-pair L22 last-prompt-token Gaussian-KL distance (raw, no centering) under the new 500-probe held-out pool; the y-axis is bystander ΔG, the trained-minus-base log-probability shift of the marker on persona A's responses under the seed-42 substrate. Each point is one ordered persona pair (A → B); n = 156. The annotated Spearman ρ = -0.52 (p = 3.04e-12) is the rank-only summary of the same pairs — negative slope, statistically distinguishable from zero — that the length-controlled CV R² of +0.21 then folds through leave-one-source-condition-out cross-validation. The CV R² collapses to the +0.21 headline because the linear fit's intercepts mis-calibrate fold-to-fold even though the rank ordering survives; the scatter shows the latent signal the CV procedure then under-resolves.

The headline is positive but indistinguishable from zero given the variance, and two of the other bars read the same way. The seed-43 cell-fixed bar comes in at +0.37; its paired difference vs the headline reads +0.16 with a lower bound essentially at zero (-0.014). The nested-search bar comes in at +0.17; its paired difference vs the headline is -0.03 across an interval of roughly ±0.22 — indistinguishable from zero given the variance. The JS baseline at -0.28 has its own CI [-0.96, -0.23] that does not cross zero from above, but the appropriate comparator is the paired difference between L22 gauss_kl and JS: the paired difference is +0.48 in gauss_kl's favor, and the interval's lower bound dips just below zero (-0.026), so "L22 gauss_kl beats JS" reads as suggestive rather than decisive. The full-panel bar at +0.54 holds close to the parent's in-sample full-panel 0.61, but the parent itself documented that ~80% of the full-panel CV R² lift over the non-stylized panel comes from the three stylized characters — the same inflation the non-stylized restriction was designed to remove — so it does not rescue the planned cell-identity claim.

A note on what "non-stylized" means here: the 156-pair panel is every ordered pair (A → B) among the 13 non-stylized personas — the panel's 16 contexts (5 personas + 5 question framings + 1 standard chat template + the 5 register rewrites that sat inside the non-stylized subset in the parent) minus the three stylized characters (pirate captain, stand-up comedian, villainous mastermind), whose pairings are dropped. The eval is teacher-forced and probe-based: the model emits nothing — each probe yields a residual-stream activation tensor and a scalar distance between two contexts, not a completion. Nothing in the planned held-out number is on-policy behavior; it reads off a controlled-linear fit.

The held-out drop is mostly pool-driven; the selection cost is small and the seed shift cannot be separated from zero

The planned design isolated three contributors to the drop from the parent's in-sample 0.34 to the held-out 0.21: the new probe pool (POOL), the inner-argmax cell selection (SELECTION), and the seed of the training substrate (SEED). Reading them off the bars, POOL shrinkage is approximately 0.135 (the parent's in-sample 0.34 down to the held-out 0.21, computed on unrounded values), above the plan's 0.10 threshold for a contributor that matters — the single biggest of the three. One caution on its strength: this contrast is a point-estimate difference with no paired CI across pools, and the headline's own CI [-0.52, +0.41] is wide enough to be compatible with the parent's 0.34, so "pool-driven" is a statement about the point estimates only.

SELECTION inflation comes to about 0.032 (cell-fixed headline minus the nested-search bar), below the plan's ~0.05 "small" threshold; the full 1737-cell inner argmax (~1121 effective after the coverage-gap exclusion) accounts for only this 0.03 of the drop.

And on the SEED axis, the seed-43 bar lands 0.163 higher than the headline (paired-difference CI [-0.014, +0.564]; zero sits just inside the low end). The magnitude is well within the seed-42 paired CI half-width of 0.466; the seed shift is indistinguishable from zero given the variance.

Figure. Across 13 outer folds the inner argmax over 1737 predictor cells picks the parent's exact L22 gauss_kl raw last-prompt cell on one fold (that pick is also the only one inside the broader L19–L24 gauss_kl raw ridge) and a scatter of mid-stack last-prompt cells on the remaining twelve. The modal pick is last_prompt × L20 × gauss_kl × centered (four folds); MMD is picked six times across layers L16–L20; c2st twice at L26 — with the single exact pick, the partition covers all 13 folds. The family region (last-prompt × {L16–L26}) is reached on every fold, but the specific ridge identity does NOT survive leave-one-source-out instability — the inner search drifts to a different metric or centering whenever the held-out source removes the small edge L22 gauss_kl had in-sample.

The cell-pick scatter explains why selection inflation is small here: the inner search stays inside the same upper-mid-stack last-prompt family across folds (finding wildly different cells fold to fold would have inflated the inner-argmax R² without the cells generalizing), so the held-out CV R² of the inner-argmax procedure stays close to the cell-fixed headline. The inner search reaches the same family region the parent's cell lives in, but it picks the exact cell on only one fold — the specific cell identity does not survive.

The plan deviation is on the POOL axis: the plan specified 4 mixed-distribution buckets matching the parent's split, and the delivered pool has 6 buckets, with 25% of probes (125 of 500) in two new categories (specialized_technical, personal_planning) that have no analog in the parent's pool. Pool-composition drift is a live alternative explanation for the 0.135 POOL shrinkage that competes with the voice-drift-fix hypothesis. The within-pool per-bucket Spearman decomposition (4 parent-analog buckets vs the 2 new ones) was not run, so I cannot disentangle "voice-drift fix lowered the headline" from "the new buckets lowered the headline." Both stay live as POOL contributors; the headline number stands regardless of which contributor dominates, but the attribution back to "the parent's voice-drift bug was the issue" is weaker than it would be with the decomposition in hand.

The seed-43 substrate retrains converged cleanly, and the seed comparator cannot be separated from the headline at this scope

The seed-43 substrate is the cleanest bar in the forest plot because it is out-of-sample on BOTH the held-out pool and the parent's substrate-selection step. Its point estimate is +0.37 and its paired difference vs the headline reads +0.16, but the interval's lower bound sits at -0.014, so the bar is indistinguishable from the headline at this scope. The implant manipulation check is clean: 16 of 16 source-persona retrains hit on-policy diagonal emission ≥ 0.80 at the test threshold, with mean ΔG = 24.2 nats over a range of 19.1–27.5 nats. The three stylized characters (pirate captain, stand-up comedian, villainous mastermind) implant slightly less (ΔG 19.1–21.0) than the rest of the panel (24.7–27.5), which is a pattern that lives entirely inside the stylized subset the non-stylized headline excludes.

Figure. Phase B manipulation check on the second-seed retrain — all 16 source-persona LoRAs implant the marker successfully (diagonal emission ≥ 0.80) under seed 43, with the three stylized characters sitting at the lower end of the ΔG range. The y-axis is diagonal ΔG (the trained-minus-base log-probability of the marker on persona A's own responses); each bar is one source persona LoRA under seed 43. The horizontal line at ΔG = 5 nats marks a generous lower bound on "implant took"; the actual range is 19.1–27.5 nats. The three stylized characters cluster at 19.1–21.0; the rest of the panel sits at 24.7–27.5. No diagonal failures; the retrains converged and the comparator carries signal.

The pooled headline hides severe per-fold dispersion — only the rank ordering transfers

The pooled +0.21 headline is not what you would get if you averaged the per-fold R² values. Under seed 42 the per-fold R²s are [-7.3, -4.2, -10.8, -28.4, -4.8, -13.1, -10.0, -25.5, -21.3, -41.0, -8.0, -62.2, -68.6] across the 13 leave-one-source-condition-out folds. Pooled R² is +0.21 only because the residuals are rank-ordered roughly correctly after pooling; mean-of-per-fold-R² is -23.5, swamped by mis-calibrated intercepts on held-out source conditions. Across folds the intercept is mis-calibrated on every held-out class; the rank ordering of distances is what carries over. The pooled R² is the appropriate summary statistic when the linear fit's intercept varies more across folds than the rank ordering of distances does — but it makes the headline more fragile to fold definition than it would naively read.

Figure. Per-fold R² of the headline linear fit, leave-one-source-condition-out, under seed 42 — every fold is deeply negative while the pooled R² is +0.21. The y-axis is the per-fold R² of the length-controlled linear fit of L22 gauss_kl distance against ΔG when the held-out fold removes one source-condition class. Each bar is one of the 13 folds; left panel is the full range (down to -68.6), right panel zooms to a clipped view. All 13 folds sit at -4.2 or lower; the pooled R² of +0.21 comes from rank-ordering working across the merged out-of-fold residuals even though the per-fold linear fit's intercept is mis-calibrated on every held-out class. Under seed 43 the per-fold structure is qualitatively different (four folds land in [-0.244, +0.145], the rest spread from -1 to -40 with one outlier at -102.0); that comparison is described in the read paragraph below rather than in this figure.

Per-fold the seed-43 substrate is qualitatively different from seed-42 — under seed 43 four folds essentially break even, landing between -0.244 and +0.145, while under seed 42 none of the folds land anywhere near zero. Even the best seed-42 fold sits at -4.2, but the comparison cuts both ways: one seed-43 fold lands at -102.0, worse than the worst seed-42 fold (-68.6), so "less catastrophic" holds on the mean; the worst case goes the other way. Under seed 43, four folds' intercepts calibrate well enough for near-zero per-fold R²; under seed 42 none do. Thirteen folds is what the paired bootstrap has to work with, and the paired difference stays indistinguishable from zero given the variance. A reader who only sees the pooled +0.21 might assume the fit transfers fold-by-fold; the dotplot shows it does not, and the seed-43 per-fold numbers soften the picture only modestly.

Reproducibility

Parameters:

Item	Value
Base model	Qwen/Qwen2.5-7B-Instruct
Adapter	LoRA r=32, α=64, dropout=0.0, target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, use_rslora=True (inherited from #474 / #460)
Optimizer	AdamW (HF Trainer), lr=1e-5, cosine schedule, bf16 (#474 recipe)
Training shape	batch_size=4 × grad_accum=4, max_length=2048; 600 rows per cell (300 positives + 300 contrastive negatives across 5 transformations); marker = ※ (token id 83399), MarkerOnlyDataCollator(tail_tokens=0) + post-response-slot suppression on negatives
Source-training epoch	1 (loc arm; the only checkpoint this experiment evaluates)
Seeds	Seed 42 (parent #502 substrate, reused) + seed 43 (this run's substrate retrain on all 16 source LoRAs)
Held-out probe pool	500 mixed-distribution probes generated by Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) across 6 buckets; 0 exact-string overlap against three reference sources (#502 pool, #474 q_train, #474 q_test); SBERT semantic dedup at cosine ≥ 0.9
Eval rig	Teacher-forced residual-stream activation extraction, layer-22 last-prompt-token Gaussian-KL distance on PCA-16 fit; CV R² of length-partial linear fit, 2000-rep paired fold bootstrap CI
CV scheme	Leave-one-source-condition-out, 13 folds (non-stylized) / 16 folds (full panel)
Hardware	1× H100 80GB (RunPod, pod-523) for Phase B retrains + Phase C activation extraction; eval scoring CPU-only
Wall time	Phase A ~3h (probe pool generation + voice-drift audit); Phase B ~30h (16 source LoRAs, seed 43); Phase C+D ~9h (activation extraction + scoring); total ~42h vs 33h budgeted
Hydra config	n/a — parameterized via environment variables on the reused #474 trainer

Artifacts:

• Held-out probe pool: eval_results/issue_523/heldout_probes_500.json
• Voice-drift / disjointness audit: eval_results/issue_523/phase_a_audit.json
• Indirect-register rewrites (the voice-drift fix): eval_results/issue_523/class_d_rewrites_extended_v1.json
• Seed-43 LoRA adapters: HF Hub superkaiba1/explore-persona-space, subfolders adapters/i523_loc_*_ep1_seed43 (browse the adapter set)
• Seed-43 implant manipulation check: eval_results/issue_523/seed43_per_cell_implant.json
• Phase D scoring JSONs (5 forest-plot bars + summary): eval_results/issue_523/scoring/
• Figures (forest plot, headline scatter, per-fold dotplot, cell-pick tally, seed-43 implant): figures/issue_523/phase_d/
• WandB training runs (16 seed-43 source LoRAs): project superkaiba/issue_523_seed43 (n/a — specific runs/<id> not captured in the events trail)

No raw completions are stored. The eval is teacher-forced and probe-based — the model emits nothing under this rig; each probe yields a residual-stream activation tensor, the predictor reads one scalar per ordered pair from those activations, and the target ΔG matrix is reused unchanged from #474. Sample-text cherry-picks for this body come from the held-out probe pool and the indirect-register rewrites (above), not from model completions. Per-pair activation tensors (.pt files) were generated pod-side during Phase C, consumed during Phase D, and discarded with the pod — they are not stored on HF Hub because regenerating them is cheaper than archiving them.

Reuse provenance:

• Reused ΔG marker-transfer matrices from #474: eval_results/issue_474/cross_eval/loc_ep1/G_logprob_matrix.json (loc-arm epoch 1) — fit: same base model + marker + coupling recipe; the cell this experiment tests was selected against this exact substrate in the parent, so reusing it is what makes the SELECTION-isolated and POOL-isolated bars meaningful; conditions match the 16 source personas under test.
• Reused parent run's headline cell identity (last prompt × L22 × Gaussian-KL × raw centering) from #502 — fit: the planned Goal cell is by definition the parent's headline cell; the cell-identity claim is what this run is testing, so the cell is named into the design rather than rediscovered.

Compute: ~42 GPU-hours vs 33 GPU-hours budgeted; 1× H100 80GB on RunPod (pod-523, terminated after upload-verification PASS). Three plan deviations consumed the extra hours: (1) a CUDA_VISIBLE_DEVICES env-prefix bug on the Phase B parallel-wave launches required a re-launch with the prefix on the wrapper; (2) the parent's #502-blob restore for the voice-drift-fix backfill round needed a separate kickoff; (3) the EPM_PROBE_POOL_STANDALONE switch flipped the probe-pool generator into standalone mode, which suppressed the planned #406 cosine cross-check (cross-check artifacts are not present in this run).

Code: Phase A probe-pool generator: scripts/issue523_phase_a_generate_probes.py; Phase B seed-43 dispatcher: scripts/issue523_phase_b_seed43_dispatch.sh (thin wrapper around the parent's scripts/i474_phase23_dispatch.sh); marker-implant trainer: scripts/i474_phase23_train.py (reused from #474, with env-var hooks EPM_TRAIN_SEED, EPM_TRAIN_EPOCHS, EPM_HF_PATH_TEMPLATE, EPM_OUTPUT_DIR_PREFIX, EPM_RUN_NAME_PREFIX); Phase C activation extractor: scripts/issue523_phase_c_extract.py; Phase D scorer: scripts/issue523_phase_d_scoring.py; forest-plot builder: scripts/issue523_plot_forest.py; no Hydra config — the experiment is parameterized via environment variables on the reused #474 trainer. Results commit 0cb990b2cbd979cc435139bd5cc57cb86eaa6188; figures commit bb71ff173e98908caa20b40aec062c03e7cc6c2c.

One-block reproduce:

git clone https://github.com/superkaiba/explore-persona-space.git
cd explore-persona-space && git checkout 0cb990b2cbd979cc435139bd5cc57cb86eaa6188
uv sync
# Phase A — generate the 500-probe held-out pool (~3h via Claude API, CPU-only)
EPM_PROBE_POOL_STANDALONE=1 uv run python scripts/issue523_phase_a_generate_probes.py
# Phase B — seed-43 source-persona retrains (16 cells, parallel-wave dispatch on 1× H100)
EPM_TRAIN_SEED=43 EPM_TRAIN_EPOCHS=1 bash scripts/issue523_phase_b_seed43_dispatch.sh
# Phase C — residual-stream activation extraction against the held-out pool
uv run python scripts/issue523_phase_c_extract.py
# Phase D — score all five forest-plot bars + paired bootstrap CIs
uv run python scripts/issue523_phase_d_scoring.py
# Forest-plot figure
uv run python scripts/issue523_plot_forest.py