On sycophancy the residual-stream persona-distance signal lives at early-to-mid layers across both extraction points, not at the parent #502 marker-leakage recipe's late-layer last-prompt anchor — and fact transfer underperforms #494's coarse predictors even before prior control (MODERATE confidence)

Human TL;DR

Headline. the residual-stream geometry that predicted marker-leakage in #502 does not transfer at its own winning cell — on sycophancy a real signal IS there but it lives at early-to-mid layers across both extraction points, and on fact the recipe underperforms #494's coarse predictors even before any prior control.

Takeaways.

• the planned anchor cell (late-layer last-prompt Gaussian-KL) lands at near-zero correlation on sycophancy and is also outperformed by simpler predictors on fact
• on sycophancy a genuine persona-distance signal exists — absolute correlation around 0.43-0.49 — but it sits at early-to-mid layers across both end-of-system and last-prompt, not at the late-layer ridge the parent recipe won at
• the early-layer signal survives the strongest competing predictor: partialling out each bystander's own baseline tendency to cave on wrong claims leaves the correlation essentially unchanged (-0.44 at the representative cell), and geometry carries roughly six times the unique variance the base rate does
• bootstrap error bars over the bystander panel back this up: every top early-layer cell's interval stays clear of zero while the recipe-anchor cell's straddles it, and a paired test on shared resamples cleanly separates the early-layer cells from the anchor — but the top cells' own intervals overlap so much that the exact leaderboard order still isn't meaningful (the paired test gives the strongest cell only a small, consistent edge over the runner-up)
• the sycophancy signal is heterogeneous across sources — five of six sources show the predicted direction, one source sits near zero with the opposite sign
• on fact the search-best cell looks strong until you control for the bystander's own prior on the taught fact; once you do, the correlation collapses to roughly zero — and the production rerun now puts an error bar around that zero (wide, low power, but zero comfortably inside)
• the deferred fact-arm reliability correction is now in: the corrected planned anchor lands just past the 0.40 threshold on paper, but its error bar straddles zero and at best it draws even with the simple coarse predictor in the paired bootstrap comparison — so the fact verdict doesn't move
• honest read: behavior-dependent at a different cell, not at the recipe cell — the parent recipe is marker-specific at its registered geometry, and there's a different early-layer signal for sycophancy that needs its own characterization

How this updates me. I'm less confident that the specific late-layer last-prompt Gaussian-KL geometry that won on the marker is a universal persona-distance axis — neither downstream behavior clears the planned threshold at that cell, and the apparent fact-arm signal turns out to be carrying the bystander's existing prior on the fact. I'm moderately more confident that there IS persona-distance signal on the sycophancy panel — the obvious deflationary story, that the early-layer cells just re-read each bystander's own base sycophancy rate, is now tested and rejected — but it lives at a different region of the residual stream and across multiple extraction points. What would still change my mind: the prompt-string-similarity control failing, or the single-seed leakage target not replicating. The honest framing is "different behavior, different cell, different extraction points" rather than "the residual stream encodes one universal persona-distance axis."

(First pass — Thomas refines this before sending to the mentor.)

TL;DR

Motivation

#502 found that one specific cell in a roughly 1500-cell predictor sweep — last-prompt × layer-22 × Gaussian-KL on an L19-L24 ridge — predicted marker-leakage transfer between personas at correlation around 0.79 on the cleanest training checkpoint. The natural next question is whether that residual-stream geometry tracks persona distance for the behavior the marker proxies for, or just for the marker itself. Two prior nulls pointed at the second answer: #494 found pooled persona-distance predictors null on fact-teaching adapters, and #470 found a single-layer cosine plus an output-distribution JS null on the #411 sycophancy panel. The critical gap was that both prior nulls used the coarse predictor family — exactly the family the parent recipe showed gets beaten by the full layer × metric × extraction-point bake-off. This run closes that gap: it takes the parent recipe, points it at the two non-marker behaviors using the already-trained adapters and stored leakage matrices, and asks whether the geometry that won on markers also wins on facts or sycophancy. Three outcomes registered, all informative — generalizes (the recipe is a behavior-agnostic persona-distance axis), marker-specific (the parent signal is specific to the marker, and the prior nulls were not resolution artifacts), or behavior-dependent (one arm yes, one arm no — tells us which axis of behavior the geometry tracks).

What I ran

A single Qwen-2.5-7B run, no retraining. For each arm I ran the existing 28-layer × 9-cloud-metric × 3-extraction-point bake-off across that arm's persona panel under each persona's verbatim system prompt, computed 9 pairwise cloud-distance metrics plus a next-token JS baseline at every layer-extraction-point cell, and scored each cell's distance matrix against that arm's stored leakage matrix with a length-controlled correlation, plus a within-stratum permutation null and a 5000-rep cluster bootstrap CI.

• Fact arm — 9 personas across 5 training substrates (marine biologist, Zelthari scholar, Qwen default, local historian, local resident, assistant, software engineer, kindergarten teacher, no-system control). The leakage target is the 26-cell stored matrix from the fact-leakage line of work (per-(teach → bystander) leakage rate, 3-seed averaged). Probe pool: 500 mixed-distribution probes inherited from the parent.
• Sycophancy arm — 24 personas across the 6-source × 23-bystander panel from the sycophancy line of work (138 off-diagonal cells). The leakage target is the per-(source, bystander) Δ from that line's frozen leakage matrix (10 rollouts × 50 probes × Claude Haiku judge per cell, single seed 42). Probe pool: 50 held-out wrong-claim probes from the same parent.
• Three comparison anchors carried into every figure: (1) the coarse predictor family from the prior fact-arm null result on the same pairs, so any bake-off cell's lift is measured against the coarse cell, not against zero; (2) the parent recipe's full-panel marker correlation of -0.75 on 240 pairs; (3) the parent recipe's non-stylized correlation of -0.58 on 156 pairs — the apples-to-apples reference, since neither downstream panel contains the three stylized characters (pirate, comedian, villain) that contributed roughly 80% of the parent's lift.

The cell I focus on is the parent recipe's winning cell — the planned anchor that this run is supposed to either replicate or refute. The search-best cell is the cell within each arm's roughly 950-1065 non-saturated scored candidates with the largest absolute correlation; it's what this arm's bake-off would have picked on its own, reported with the explicit "selected from 950+ candidates" caveat. The ridge mean is what doesn't depend on a single lucky cell.

The fact arm's main sweep ran in smoke mode because the per-seed reliability reconstruction that production-mode statistics need was not implemented at sweep time (a known-missing feature, not a bug). The smoke fallback pins the reliability factor at 1.0, so the sweep-level fact numbers carry no attenuation correction; the relative ordering across cells is invariant under a shared-reliability assumption. A third zero-GPU follow-up later closed this gap at the two fact headline cells — described below, results in the final finding. The sycophancy arm ran in production mode end to end.

One zero-GPU follow-up analysis ran on the sycophancy arm after the main bake-off: I re-scored the 29 early-layer cells (the layers-0-13 cosine cluster across both extraction points, plus the layer-7 last-prompt MMD global max) against the same 138-cell leakage matrix with each bystander's intrinsic base sycophancy rate as a competing covariate — source-controlled partial correlations in both directions plus a variance decomposition, on the full panel and on the live-cell subset, with the same 5000-rep bootstrap and 2000-rep permutation machinery the production arm used. No new activations, no training, no generation — pure re-analysis of stored matrices.

A second zero-GPU follow-up put error bars on the resulting leaderboard: 5000 bootstrap resamples of the bystander panel (the complement of the source-clustered intervals the production stats used) on the top-15 sycophancy cells plus the recipe-anchor cell, paired cell-vs-cell differences computed on shared draws, and a two-way source-and-bystander sensitivity on the two strongest cells — again pure re-analysis of the stored matrices.

A third zero-GPU follow-up closed the fact arm's smoke gap at its two headline cells. I reconstructed per-(teach → bystander, seed) leak rates from the per-seed judged completions already on the HF data repo — 18 of the 26 cells re-scored from the stored Claude Haiku judge labels, 4 from git-tracked judge aggregates, and 4 Zelthari-scholar cells via a substring scorer that feeds only the variance/reliability path (the point estimates stay the frozen 3-seed CSV values) — cross-checked the reconstruction against the frozen target matrix (worst judge-cell disagreement: one judge flip in 180 completions), computed the reliability factor the smoke run had pinned at 1.0 (it comes out at 0.78 panel-wide), and re-scored the planned anchor and the fact search-best cell with attenuation adjustment plus a substrate-clustered bootstrap and permutation test. No new generations, no new judge calls — pure re-analysis of stored labels.

Three cherry-picked illustrative pairs from the sycophancy bake-off's early end-of-system cosine cell, source = software engineer (one of the strongest sources in this cell's per-source spread). Full distance matrix at the link in the dropdown.

Bystander	centered cosine distance	actual leakage Δ (trained-base)	predictor reads
data scientist	0.41	+0.60	short distance, biggest leakage in the panel — top-1
comedian	0.86	+0.48	mid distance, mid-high leakage — the "surprise" bystander the prior coarse predictors ranked 23rd of 23
assistant	1.25	-0.03	long distance, no leakage — the recipe's largest distance in this source row

The pattern at this cell on this source is monotone: shorter early-layer end-of-system cosine distance from the software-engineer source predicts larger Δ in the leakage matrix, including ranking the comedian 5th of 23 instead of the 23rd-of-23 placement every coarse predictor in the prior null produced. The same pattern is NOT uniform across sources — see the per-source breakdown in the second finding. Full pair-by-pair predictor values at the metric file on HF data; full leakage target at syco_411_analyze_summary.json.

This run generates no model completions — every scored cell is one pairwise-distance scalar over already-captured base-model residual-stream activations. The measurement-validity tell is that the body cannot show "the model said X" examples; the predictor reads as one number per persona pair per cell, and the only thing the bake-off learns is whether that number tracks the leakage target.

Findings

At its own anchor cell and at the late-layer ridge, the parent recipe does not generalize to either behavior

The first question is the cleanest one: does the parent recipe's winning cell still win on either downstream arm? The figure puts the parent's two reference correlations on the marker panel alongside the same cell measured on this run's fact and sycophancy panels, plus the mean across the 18-cell late-layer ridge for each arm. The dashed line is the planned threshold the registration trigger asked for.

Figure. The marker-leakage predictor cell does not generalize to fact or sycophancy leakage at the planned anchor or at the late-layer ridge mean. Each bar is the absolute correlation between that cell's pairwise distance matrix and the corresponding arm's leakage target. Solid bars = single-cell anchors; hatched bars = ridge means across the 18 cells in the late-layer (L19-L24) last-prompt × {Gaussian-KL, MMD, Wasserstein-2} band. The two parent reference bars are pulled from the parent recipe's regression output; the four downstream bars are the anchor cell + ridge mean per arm (fact = smoke mode; sycophancy = production with 5000-rep cluster bootstrap CI clustered on source). The sycophancy anchor's error bar straddles zero. The dashed grey line is the planned threshold of 0.40. Neither downstream arm clears it at the anchor cell or at the ridge mean.

On sycophancy, the planned anchor cell comes back null on every leg of the registered trigger: absolute correlation is 0.09, the within-source permutation p-value is 0.30, and the cluster bootstrap 95% CI straddles zero. The late-layer ridge mean across {Gaussian-KL, MMD, Wasserstein-2} has a signed mean of -0.02 across 18 cells with zero of the 18 clearing 0.40 — not one ridge cell, not a near miss. The strict reading is "marker-specific at this cell on this panel," with the binding constraint that the comedian recovery rank at the anchor is 23rd of 23 (the same failure mode the prior coarse predictors hit).

One methodology caveat the reader is owed. At the same cell the pooled correlation (without source-controlled residualization) is -0.40 — in the marker-leakage direction, at about half the parent's pooled magnitude. The parent recipe used pooled correlation on a 16-condition marker panel without source clustering; this run adds source-controlled residualization on a 6-source × 23-bystander panel because the within-source clustering is strong enough to confound pooled estimates. The source-controlled step is doing real work — across the 814 sycophancy cells whose pooled correlation exceeds 0.10 in magnitude, 243 have opposite signs for the pooled and source-controlled estimates. So the sycophancy anchor verdict is more precisely "the pooled association is in the marker direction at smaller magnitude, but disappears once source clustering is partialled out." Both numbers belong in the body; I default to the source-controlled number for the threshold comparison because the source clustering on this panel is real and is what the run's registration anticipated.

The fact arm's number at the same cell looks larger (0.36) but is misleading in three ways. First, smoke mode pins the reliability factor at 1.0, so the 0.36 is the un-corrected number, not the attenuation-adjusted number the registered threshold asks for — the production rerun in the final finding puts the adjusted number at +0.41, nominally past the threshold, but with an error bar that straddles zero. Second, the sign is positive — the wrong direction for "closer personas leak more" relative to the parent's marker anchors at -0.58 and -0.75. Third, the bake-off cell's correlation on the same 25 pairs underperforms the coarse fact-slice JS predictor by about 0.25 correlation points (raw 0.36 vs coarse 0.59). So even before the ridge mean (0.08 on 18 cells, similar story), the fact arm fails to clear the lift-over-coarse leg of the registered trigger at the cell that was supposed to be the universal winner.

On sycophancy a real signal lives at early-to-mid layers across both extraction points — but it is NOT the parent recipe

The first finding said the parent's specific cell is null on sycophancy. The second question is whether the sycophancy arm has signal SOMEWHERE in the predictor zoo. The figure faceted by arm (top = fact, bottom = sycophancy) and by metric family (left = cosine, right = Gaussian-KL) traces the per-layer correlation for each extraction point. The grey band marks the parent's winning ridge; the dotted and dashed reference lines are the registered threshold and the parent's non-stylized anchor.

Figure. Per-layer correlation with leakage for the two metric families that mattered, faceted by arm and by extraction point. Rows: fact arm (top, n=18 per cell, smoke mode) and sycophancy arm (bottom, n=138 per cell, production stats). Columns: cosine (the early-layer surprise; left) and Gaussian-KL (the parent's winning metric family; right). Three extraction-point lines per panel: blue = end of system prompt, orange = last prompt token (the parent's extraction point), green = mean over response tokens. The grey vertical band marks layers 19-24 (the parent's winning ridge); shaded bands around the sycophancy lines are 5000-rep cluster bootstrap 95% CIs (clustered on source). Two reference horizontals: dotted at -0.40 (the registered threshold on the negative side, since the sycophancy signal is negative), dashed at -0.58 (the parent's non-stylized anchor). The bottom-left panel is the story: on sycophancy, early-to-mid layer (L0-L13) signal lives at BOTH end-of-system cosine (blue, correlations around -0.43 to -0.49 at L0-L8 with CIs clear of zero) AND last-prompt cosine (orange, around -0.42 to -0.45 at L5-L8 with CIs clear of zero). The bottom-right panel shows the recipe's specific metric family is flat at every extraction point through the late-layer ridge on sycophancy.

The strongest single cell on the sycophancy bake-off is an early last-prompt cell at absolute correlation 0.499 (within-source permutation p = 0.0005, the floor of the 2000-rep null). The cell I built the cherry-picked example around — a layer-2 end-of-system cosine cell at absolute correlation 0.489, same permutation p, comedian recovery rank 5th of 23 — is the second-best by a hair and is the representative max of a tightly correlated cluster spanning layers 0-13 across both end-of-system and last-prompt extraction. The signal is not a single lucky cell: 172 of 1065 scored cells clear permutation p ≤ 0.001 (94 last-prompt, 74 end-of-system, 4 mean-response), and 9 of the top-15 cells by absolute correlation are end-of-system while 6 are last-prompt. End-of-system dominates the very top of the leaderboard while last-prompt dominates the wider p ≤ 0.001 band — both extraction points carry signal, with the balance flipping depending on whether you read the top-15 or the broader significant band. On the live-cells-only subset (the 21 cells where the leakage target's Δ exceeds 0.10 in magnitude, restricting to where the target has real variance instead of binomial noise floor), the layer-2 end-of-system cosine cell holds at absolute correlation 0.473 on 21 cells.

Per-source heterogeneity at the layer-2 end-of-system cosine cell: the per-source correlations are -0.76 (comedian), -0.63 (software engineer), -0.54 (assistant), -0.49 (villain), -0.30 (Qwen default), and +0.05 (kindergarten teacher). Five of six sources show the predicted negative association, but the kindergarten-teacher source sits near zero with the wrong sign. So the cell-level 0.489 is a weighted average of strong-signal sources and one near-null source — the signal is heterogeneous, not a uniform persona-distance axis. This matters for downstream extrapolation: a result obtained on a panel with more kindergarten-teacher-like sources may show a smaller effective correlation even if the cell really IS picking up persona distance.

The honest reading is behavior-dependent at a different cell, not at the recipe cell. The residual stream IS carrying persona-distance signal that tracks sycophancy leakage on this panel, but the signal sits in early-to-mid layer geometry across both end-of-system and last-prompt extraction points rather than in the late-layer last-prompt cloud-metric ridge that won on markers. The recipe's specific contribution — late-layer cloud-aware metrics on the last prompt token — doesn't add anything to this signal: the right column shows the Gaussian-KL family flat across every extraction point through the late-layer ridge on sycophancy. The early-to-mid layer cell IS selected from roughly 1065 scored candidates, so the absolute-correlation numbers carry the selection-inflation caveat; the honest defense is the L0-L13 cluster structure (14+ cosine cells across two extraction points all clearing perm p ≤ 0.001 in the same direction) plus the comedian-recovery diagnostic. Two mundane alternatives needed ruling out. The first — that the early-layer cells just re-read each bystander's intrinsic willingness to cave on wrong claims, the strongest known competing predictor on this panel — is tested and rejected in the next finding. The second, not yet ruled out: at very early end-of-system layers (L0-L2), the residual is only minimally transformed from the embedding of the system prompt, so the centered cosine distance could be partly tracking how different the system-prompt STRINGS are rather than what the model has done to them — the fact that the same signal magnitude appears at last-prompt layer 7 (well above the embedding layer, past the user turn) helps rule this out, but a token-overlap control on the persona system prompts would close it.

The early-layer signal is not the bystander's base sycophancy rate in disguise

The strongest competing predictor on this panel is not geometric at all: each bystander has its own intrinsic willingness to cave on wrong claims before any training touches it, and that base rate has been the dominant predictor everywhere it has been tested against geometry on this line of work. If the early-layer cosine signal were just re-reading "agreeable personas leak more," the previous finding would deflate to a restatement of the base rate. A zero-GPU follow-up re-scored all 29 early-layer cells (the layers-0-13 cosine cluster across both extraction points plus the layer-7 last-prompt MMD global max) with the bystander's base sycophancy rate as a competing covariate on the same frozen 138-cell panel.

Figure. Partialling out the bystander's base sycophancy rate leaves the early-layer geometry signal essentially untouched. Solid lines = source-controlled partial correlation (geometry given base rate) per layer on the 138-cell panel, trained-minus-base leakage target; faint dashed lines = the same cells' geometry-alone correlation — the two sit on top of each other at every layer. Blue = end of system prompt, orange = last prompt token (cosine, layers 0-13); black diamond = the layer-7 last-prompt MMD global-max cell. Dotted horizontal = the base rate alone (−0.18). Error bars = 5000-rep source-clustered bootstrap 95% CIs on the two headline cells: layer-2 end-of-system cosine partial −0.444, CI [−0.608, −0.252]; layer-7 last-prompt MMD partial −0.538, CI [−0.657, −0.314]; both with within-source permutation p < 10⁻³ on n = 138.

The signal survives essentially unattenuated. At the representative layer-2 end-of-system cosine cell the geometry partial holds at −0.44 after removing the bystander's own base sycophancy rate — the geometry-alone correlation under this convention is also −0.44, so the partial loses nothing — while the base rate alone reads −0.18 and the reverse partial (base rate given geometry) sits at −0.21. The variance decomposition makes the asymmetry plain: geometry uniquely explains about 19% of within-source variance, the base rate about 3%.

Across all 29 cells the partial sits within 0.03 of the raw correlation — there is no cell where the base-rate control bites. The same holds where the base rate enters non-circularly: on the trained leakage rate (rather than the trained-minus-base difference, which mechanically contains the base rate), the base rate alone is a real predictor at +0.39, yet the geometry partial still reads −0.48. On the live-21 subset the partials are larger (−0.59 cosine, −0.63 MMD), but the live cells sit in only 2 of 6 sources, so the source-clustered bootstrap CIs there are degenerate — those two numbers are directional only, and the citable inference is the full-panel CI.

Two reading notes: the point estimates here use the rank-then-demean convention of the earlier base-rate analysis without attenuation adjustment, so the −0.44 raw number IS the same cell that reads −0.489 in the previous finding's attenuation-adjusted convention — a convention difference, not drift. And the per-source spread persists under the partial (strongest source −0.76, weakest −0.15), so the heterogeneity caveat from the previous finding carries over unchanged.

This analysis is predictor-only — no model completions exist for it; each of the 29 cells contributes one partial-correlation read over the already-stored distance matrices and the frozen leakage panel. Full per-cell numbers, per-source partials, and the live-21 degeneracy flag: baserate-partial-early-layer-cosine/results.json. For context on how much the covariate swap matters: the same partial analysis with the previously-registered single layer-20 cosine as the geometry covariate had left geometry with essentially nothing (partial +0.08, unique variance under 1%) — swapping in the early-layer cells is what flips geometry from useless-given-the-base-rate to carrying the unique variance.

Resampling the bystander panel separates the top cells from the recipe anchor — and the paired test, not the marginal error bars, splits the top two

The second finding left an obvious loose end: the top of the sycophancy leaderboard is a tightly correlated cluster whose absolute correlations sit within about 0.1 of each other, so are the similar-looking scores statistically separable at all, or is the leaderboard order pure noise? This follow-up puts error bars on the leaderboard by bootstrap-resampling the bystander panel — the 24 distinct bystander personas drawn with replacement while keeping all 6 sources, the complement of the source-clustered intervals the production stats reported — on the top-15 cells plus the recipe-anchor cell, and then asks the cell-vs-cell question directly with paired differences computed on shared draws (three pairs: the two strongest early-layer cells against each other and each against the anchor — not every top-15 cell versus the anchor).

Figure. Every top leaderboard cell's bystander-resampled interval excludes zero while the recipe anchor's straddles it, and the paired shared-draw test separates the global-max cell from both the anchor and the runner-up. Left: percentile-bootstrap 95% CIs (B = 5000, seed 42, n = 138 cells per estimate) on the source-controlled Spearman ρ for the 12 distinct top-15 leaderboard cells plus the recipe-anchor cell (green); thick bars resample the 24 bystander personas with replacement, thin bars resample the 6 sources. Global max (last prompt token, layer 7, MMD): ρ = −0.522, bystander CI [−0.629, −0.377]. Representative early cosine (end of system prompt, layer 2): ρ = −0.436, CI [−0.532, −0.309]. Recipe anchor (last prompt token, layer 22, Gaussian-KL): ρ = +0.008, CI [−0.234, +0.284]. Right: 95% CIs of the paired difference in absolute ρ on shared bystander draws — global max minus early cosine = +0.087 [+0.011, +0.154]; global max minus anchor = +0.515 [+0.149, +0.569]; early cosine minus anchor = +0.428 [+0.078, +0.476]. All three paired differences exclude zero.

On the bystander axis the picture is two-tier. Every one of the top-15 leaderboard cells keeps its interval clear of zero — the weakest still tops out at −0.26 — while the recipe-anchor cell straddles zero on this axis too, just as it did under source resampling. But the marginal intervals of the top cells overlap heavily on every resampling axis, so reading the leaderboard order off the point estimates remains as unjustified as the second finding warned.

The paired test on shared draws is the right instrument for cell-vs-cell claims, and it splits all three gaps it was pointed at: the global-max MMD cell beats the early cosine cell by +0.09 in absolute correlation, ahead in 98.6% of resamples, and both early-layer cells beat the anchor by +0.43 to +0.51 — the dominant separation, five to six times the size of the top-two gap. For the remaining top cells, which got no paired test, the separation from the anchor rests on the disjoint marginal intervals — every top cell's interval sits entirely below −0.25 while the anchor's reaches down only to −0.23. Resampling sources and bystanders simultaneously, the most conservative axis, still leaves both early-layer cells clear of zero.

The small top-two gap is a cell-vs-cell statement, not a metric-family ranking: the two cells differ in metric (MMD vs cosine) AND extraction point (last prompt vs end of system) AND layer (7 vs 2), so the paired result says only that the global-max cell's edge over the representative early cosine cell is consistent within resampled panels — under one estimator convention, on one panel, against the single-seed leakage target. And the selection-inflation caveat on the global max (picked from roughly 1065 scored cells) is not erased by a paired interval computed at the same selected cell.

One reading note: the point estimates here use the rank-then-demean unadjusted convention of the base-rate follow-up, so the −0.522 and −0.436 in the figure are the SAME two cells that read −0.499 and −0.489 in the second finding's attenuation-adjusted convention — a convention difference, not drift; the runtime cross-check against the base-rate follow-up's geometry-alone values agreed exactly. Like the base-rate control, this analysis generates no completions — each bootstrap draw re-scores the stored distance and leakage matrices.

On fact, the bake-off underperforms the coarse predictors and collapses under the bystander-prior control

The third question is whether the fact arm has signal somewhere — not at the planned anchor (we already know it doesn't), but anywhere. The figure puts the search-best cell alongside the coarse predictors from the prior fact-arm null result computed on the SAME 18-pair slice, plus the same search-best cell after partialling out the bystander's own prior on the taught fact.

Figure. On the fact arm, the bake-off underperforms the coarse predictors at the planned anchor, and even the search-best cell collapses to roughly zero once the bystander's own prior on the fact is partialled out. Bars 1-5 are the coarse predictors from the prior fact-arm null result (two cosine variants at layer 21, next-token JS on-topic, fact-slice JS, and the bystander's own log-prob on the fact) computed on the SAME n=18 source-controlled cells as the search-best (bar 7), pulled from that cell's own per-coarse correlation field. Bar 6 = the planned anchor (last-prompt layer-22 Gaussian-KL, absolute correlation 0.36, n=25 cells; this bar uses a different denominator because the anchor cell has 25 valid pairs while the search-best has 18). Bar 7 = the bake-off's search-best cell on the fact arm (end-of-system layer-1 cosine; layers 1-4 are tied at this same 0.67 since early-layer cosine is essentially a rank-preserving function of the embedding). Bar 8 = the same search-best cell with both substrate AND the bystander's prior on the fact partialled out (0.03). The bake-off's win at the search-best cell is entirely the prior; once you control for what the bystander already knew about the fact, there's nothing left to explain.

Two failures stack on this arm. At the planned anchor, the bake-off cell reaches raw correlation 0.36 on the 25 pairs where the cell has variance, but the same 25 pairs fit the coarse fact-slice JS predictor at 0.59 — the bake-off cell's lift over the coarse predictor is -0.25. Across the entire fact arm, zero of 953 scored cells beat the coarse maximum by the required 0.15 correlation points. The recipe's cloud-aware metric on the last prompt token is a worse predictor of fact transfer than the prior baselines at its own planned anchor.

At the search-best cell (end-of-system layer-1 cosine; layers 1-4 are tied because early-layer cosine is essentially a rank-preserving function of the embedding), the source-controlled correlation reaches 0.67 — a number that LOOKS like it clears the planned threshold of 0.40. The collapse happens at the next step. Once you partial out both substrate AND the bystander's own log-prob on the taught fact (the dominant predictor of fact leakage that a prior result already established), the double-controlled correlation drops to 0.03. The 0.67 was carrying the prior, not orthogonal-to-the-prior persona-distance signal. The registered conditions table explicitly built this double-controlled comparison as the test for "does the bake-off lift OVER the prior" — the answer at the search-best cell is no.

A handful of OTHER non-saturated cells do retain double-controlled correlations around 0.37-0.40 after the prior partial. The 9 surviving cells are spread across last-prompt and mean-response extraction points and four metric families. With 953 scored fact cells the multiple-comparisons floor is permissive, and unlike the sycophancy signal these cells don't form a tight cluster and lack a recovery-style sanity check — so they lack the structural defenses the sycophancy signal has against selection inflation. The honest verdict is "9 cells survive the prior partial at moderate correlation from a 953-cell search, but the same multiple-comparisons discipline that anchors the sycophancy reading also disciplines these — the structural evidence is not there." The headline holds in the qualitative direction (the bake-off does not robustly beat the prior on fact) while explicitly not claiming the bake-off cleanly fails everywhere.

The smoke caveat, and what happened to it: at sweep time the fact numbers carried no attenuation correction, and I flagged that a production rerun could push the planned anchor's 0.36 past 0.40 and would let me put a cluster bootstrap CI on the 0.03 collapse. That rerun has now happened — next finding. The short version: the ordering claim survives exactly as predicted (attenuation scales the bake-off cells and the coarse predictors by the same shared factor, and the anchor's paired lift over the coarse fact-slice JS predictor tops out at exactly zero in the substrate-clustered bootstrap — at best a draw, ahead in at most a small tail of resamples), while the absolute-threshold story turned out to be exactly the edge case flagged here — the adjusted anchor lands at +0.41, nominally past the threshold, with a CI that straddles zero. One diagnostic remains un-run: the prior-only residualization (the bystander's prior partialled out WITHOUT substrate fixed effects; substrate + prior is the double-controlled estimate, prior-only is not stored), so a small piece of the collapse could still be over-residualization that a prior-only test would catch.

A small replication note on the apparent tension with the prior fact-arm null result: that prior result reported a pooled-predictor null on fact, while this run shows same-pair coarse fact-slice JS reaching 0.59 on the source-controlled 25-pair slice. This is NOT a contradiction. The prior null is on the pooled estimate across heterogeneous training substrates; this run reports the same-cell coarse predictor AFTER substrate-controlled residualization on the same restricted set of pairs the bake-off cell sees. The two numbers answer different questions; the within-substrate same-pair correlation is what the bake-off cell needs to beat, and it does not.

Reliability-corrected, the fact anchor nominally crosses the threshold — but its CI straddles zero and it still never beats the coarse predictor

The fact arm's smoke mode left three open questions: does attenuation correction push the planned anchor past the registered 0.40, is the search-best cell's collapse-to-zero under the prior control measurable or just un-measured, and does the bake-off-loses-to-coarse ordering survive production statistics. The zero-GPU rerun described in the setup answered all three by reconstructing per-(teach → bystander, seed) leak rates from the per-seed judged completions on HF and re-scoring the two headline cells in production mode — attenuation adjustment, substrate-clustered bootstrap with 5000 draws, 2000-rep within-substrate permutation null, and paired same-draw comparisons against each coarse predictor. The figure shows both cells under three estimates each: the raw smoke-mode number, the reliability-corrected number, and the bystander-prior-partialled number.

Figure. Attenuation lifts the planned anchor past the registered threshold on point estimate, but every interval that matters straddles zero. Signed substrate-partialled Spearman correlation with fact leakage for the two fact headline cells, three estimates each: raw (the smoke-mode number), attenuation-adjusted (reliability-corrected, panel reliability 0.78), and bystander-prior partialled. Error bars = substrate-clustered bootstrap 95% CIs (5000 draws, seed 42); dashes mark the registered threshold at ±0.40. Planned anchor (last-prompt layer-22 Gaussian-KL, n = 25 pairs): raw +0.364 → adjusted +0.411, CI [−0.010, +0.746], permutation p = 0.085. Search-best (end-of-system layer-1 cosine, n = 18 pairs): raw −0.668 → adjusted −0.737; prior-partialled −0.026, CI [−0.785, +0.714].

The reconstruction itself checked out cleanly: the rebuilt per-seed rates agree with the frozen target matrix at worst one judge flip per 180 completions on the 22 judge-scored cells, and the rebuilt raw correlations reproduce the stored sweep values exactly at both cells. The reliability factor comes out at 0.78 panel-wide (0.787 at the anchor, 0.821 at the search-best cell), which lifts the anchor's +0.364 to +0.411 — nominally past the registered 0.40. But the crossing is a point estimate, not a detection: the substrate-clustered interval dips just below zero and the permutation p is 0.085, so the corrected anchor remains statistically indistinguishable from zero on this panel.

And the second leg of the registered trigger fails outright: on shared bootstrap draws the anchor's paired lift over the coarse fact-slice JS predictor is −0.333, with an interval that tops out at exactly zero — at best the anchor draws even with the coarse predictor, coming out ahead in at most a small tail of resamples. The fact verdict from the previous finding therefore stands, now with error bars on it.

The double-controlled collapse at the search-best cell is confirmed, but at very low power: the point estimate lands at −0.026, with zero sitting comfortably inside an interval far too wide to pin anything down. The interval is enormous because the double-controlled estimate survives on only 18 pairs across 3 training substrates — resampling 3 clusters with replacement yields at most 10 distinct cluster multisets, so the bootstrap quantiles are coarse. This is honest low power, not evidence of absence: the interval cannot rule out large effects in either direction, and the citable claim is "the prior control removes the point estimate," not "the prior control proves zero."

Three smaller reading notes. The 4 Zelthari-scholar cells lack persisted per-seed judge labels for two of the three seeds, so a substring scorer stood in on the variance path only; calibrated against the one seed where both scorers exist, it runs a consistent +0.08 above the judge across all four question frames — a level offset that biases neither the variance estimate nor the point estimates, which stay the frozen CSV values. The attenuation adjustment treats the reliability factor as a known constant, so adjusted CI bounds can exceed 1 in magnitude (the search-best adjusted lower bound is −1.02); I report them unclipped rather than masking the artifact.

And this analysis generates no completions — it reads stored judge labels and re-scores stored matrices, so there is nothing new to quote; the underlying per-seed judged completions are linked in the reproducibility section.

Reproducibility

Parameters:

field	value
base model	Qwen/Qwen-2.5-7B (BF16, single seed 42, greedy decoding)
adapter	none (no retraining; activations captured from the base model under each persona's verbatim system prompt)
fact-arm panel	9 personas, 26 cells across 5 training substrates, cond_ids FB1..FB9; verbatim system prompts copied from #444 + #192 + #381 / #389 / #390 prompt registries
syco-arm panel	24 personas, 138 off-diagonal cells (6 sources × 23 bystanders), cond_ids SC1..SC24; verbatim system prompts copied from #411's per-source panel_prompt field
extraction points	end of system prompt, last prompt token, mean over response tokens
layers	all 28 residual-stream layers (0-27), pre-norm residual via forward hooks on model.model.layers[L]
metrics	cosine, Euclidean, Mahalanobis, pooled-Mahalanobis, MMD (unbiased), classifier-two-sample-test (5-fold AUC), spectral delta, Gaussian-KL on PCA-16 fit, Bures-Wasserstein-2; plus the next-token JS baseline on the full vocabulary at the last prompt position
fact target	per-(teach → bystander) leak_rate from eval_results/issue_494/regression_data.csv, 3-seed averaged (seeds 42 / 137 / 256), 26 cells across 5 substrates
syco target	per-(source, bystander) Δ from eval_results/issue_480/_inputs/syco_411_analyze_summary.json (#411 frozen leakage matrix, 10 rollouts × 50 probes × Claude Haiku judge per cell, single seed 42)
fact probe pool	500 mixed-distribution probes from eval_results/issue_502/probes_500.json
syco probe pool	50 held-out wrong-claim probes from data/issue_509/eval_50_probes_for_extraction.json (reformatted from #411's eval_50.jsonl)
scoring	length-partial Spearman ρ (pooled, substrate-FE-residualized on fact, source-FE-residualized on syco, double-FE on fact for #500-prior control); LOCO-CV R²; 5000-rep cluster bootstrap CI; within-stratum permutation null (B = 2000 production / B = 50 smoke); delete-one-source jackknife on syco; attenuation-adjustment via reliability_y
smoke vs prod	fact arm = SMOKE (B_perm = 50, B_bootstrap = 0, reliability_y = 1.0 fallback because per-seed SE reconstruction is NOT implemented; documented missing feature); syco arm = PRODUCTION (B_perm = 2000, B_bootstrap = 5000)
sweep scale	1513 scored metric files per arm (28 layers × 3 extractions × 9 metrics × 2 variants = 1512 + next-token JS baseline). Saturation screener marks n_cells_skipped_na = 560 (fact) / 448 (syco), leaving 953 / 1065 scored cells. Inside the scored set, 561 of 953 fact cells exclude at least one per-pair saturated entry
n per panel	fact arm: 26 cells pooled / 18 substrate-FE-residualized / 25 cells at the planned anchor after per-pair saturation exclusion; syco arm: 138 off-diagonal cells; 21 live-cells subset (
methodology shift vs #502	this run adds source-FE on syco and substrate-FE on fact, which the parent #502 did NOT use (parent reported pooled length-partial Spearman on a 16-condition marker panel without source clustering). Across 814 syco cells with `
anchor cell stat (syco, full)	last_prompt × L22 × gauss_kl × centered: rho_fe_adj = +0.089, perm_p_fe = 0.302, n = 138, cluster bootstrap 95% CI [-0.267, +0.345]; rho_pooled_adj = -0.403
anchor cell stat (fact, full, smoke)	last_prompt × L22 × gauss_kl × centered: rho_fe = rho_fe_adj = +0.364, n = 25 substrate-FE cells, coarse_lift.delta_rho = -0.252 vs fact_slice_js baseline
sycophancy global max	last_prompt × L07 × mmd × centered: rho_fe_adj = -0.499, perm_p_fe = 0.0005, 95% CI [-0.609, -0.332], n = 138
representative early end-of-system cell	end_of_system × L02 × cosine × centered: rho_fe_adj = -0.489, perm_p_fe = 0.0005, 95% CI [-0.605, -0.284], n = 138; live-cells-only rho_fe_live = -0.473 on 21 cells; comedian recovery rank 5/23; per-source ρ = {comedian: -0.762, software_engineer: -0.631, assistant: -0.538, villain: -0.492, qwen_default: -0.304, kindergarten_teacher: +0.047}
fact search-best cell	end_of_system × L01 × cosine × centered: rho_fe = -0.668, rho_double_fe = -0.026, n = 18 (L1-L4 tied at the same value)
code-fix landings during run	4 surgical fixes that advanced the pipeline without altering recipe or panels: (1) f29381dbe per-pid tmp suffix in _write_json_atomic (race on shared meta.json.tmp); (2) a93fdd74e system_prompt=None NoneType crash in _build_prompts_for_extraction for bare-Qwen conditions; (3) 0aac536f5 merge-step partition AssertionError on system_prompt=None Class A conditions; (4) 8476202bc AttributeError on matrix:null N/A cells from the saturation screener — returns None + counts n_cells_skipped_na + ≥25% sanity-cap guard
Hydra / driver	scripts/issue493_extraction_metric_bakeoff.py driver under scripts/issue502_dispatch.py --phase metrics; scoring in scripts/issue509_scoring.py (new, ~700 LOC); --conditions-registry + --probe-pool-mode flags added to bypass the q_test prefix gate for the syco arm
hardware	1× RunPod pod-509, 8× H100 80GB SXM5, terminated 2026-06-08T06:23:55Z after upload-verification PASS
wall time	~6.5 hours end-to-end on 8× H100

Follow-up parameters (base-rate control on the early-layer cells, label baserate-partial-early-layer-cosine):

field	value
cells scored	29 — cosine × centered at layers 0-13 × {end of system, last prompt} (28 cells) + last_prompt × L07 × mmd × centered (the global max)
targets / panels	trained − base (primary) + trained rate (secondary); all-138 panel + live-21 subset (live cells sit in 2 of 6 sources — live_21_bootstrap_degenerate: true in the JSON; live-21 CIs are not citable inference)
statistics	source-FE Spearman, source-FE partial Spearman (geometry given base rate AND base rate given geometry), within-R² decomposition; point-estimate helpers copied verbatim from the prior base-rate covariate script (rank-then-demean, no attenuation adjustment — NOT directly comparable to the attenuation-adjusted rho_fe_adj values above)
inference	two headline cells only: 5000-rep source-clustered bootstrap (seed 42) + 2000-rep within-source permutation (hashed-tag seeds, same-statistic null)
headline (L02 end-of-system cosine, all-138)	partial = −0.444, CI [−0.608, −0.252], perm p < 10⁻³; raw geometry-alone −0.436; base-alone −0.183; unique within-R² 0.191 (geometry) vs 0.034 (base rate)
headline (L07 last-prompt MMD, all-138)	partial = −0.538, CI [−0.657, −0.314], perm p < 10⁻³; raw −0.522
inputs	distance matrices from the HF data repo at pinned revision 1b6e20530b1c6d477a387c18d5a88554910e7df9; same #411 frozen target matrix as the main run; 0 pairs excluded by the saturation screen on all 29 cells

Follow-up parameters (bystander-resampled CIs + paired metric differences, label bystander-bootstrap-cis):

field	value
cells scored	16 — the global max (last_prompt × L07 × mmd × centered), the representative early cosine cell (end_of_system × L02 × cosine × centered), the recipe anchor (last_prompt × L22 × gauss_kl × centered), plus the top-15 leaderboard cells by attenuation-adjusted absolute correlation from the syco scoring file (12 distinct after the Euclidean centered/raw ties collapse)
resampling axes	bystander-cluster (primary; the 24 distinct bystander personas drawn with replacement, all 6 sources kept — each source's own panel is the other 23), source-cluster (6 sources, recomputed on the same unadjusted estimator rather than copied from scoring.json), two-way source × bystander (the two strongest cells only)
bootstrap	B = 5000, seed 42, percentile [2.5, 97.5] CIs, NaN guard below 100 finite reps (all reported cells: 5000 finite reps)
headline CIs (bystander axis)	global max ρ = −0.522, CI [−0.629, −0.377]; early cosine ρ = −0.436, CI [−0.532, −0.309]; anchor ρ = +0.008, CI [−0.234, +0.284]; two-way sensitivity: global max [−0.710, −0.215], early cosine [−0.650, −0.126]
paired differences (shared bystander draws)	Δ|ρ| global max − early cosine = +0.087, CI [+0.011, +0.154], global max ahead in 98.6% of draws; global max − anchor = +0.515, CI [+0.149, +0.569]; early cosine − anchor = +0.428, CI [+0.078, +0.476]
estimator convention	rank-then-demean source-FE Spearman, unadjusted — copied verbatim from the base-rate follow-up script; runtime cross-check against that follow-up's geometry-alone values (hard error on drift above 1e-9) passed, and the code review independently re-implemented and re-derived the headline CIs. The scoring file's demean-then-rank + attenuation-adjusted values are used ONLY to pick the top-15 cells
inputs	distance matrices from the HF data repo at pinned revision 1b6e20530b1c6d477a387c18d5a88554910e7df9; frozen leakage target eval_results/issue_480/_inputs/syco_411_analyze_summary.json; cell selection from eval_results/issue_509/syco_arm/scoring.json

Follow-up parameters (fact-arm production rerun with per-seed reliability reconstruction, label pathb-fact-rerun):

field	value
cells re-scored	2 — the planned anchor (last_prompt × L22 × gauss_kl × centered, n = 25) and the fact search-best (end_of_system × L01 × cosine × centered, n = 18); statistics computed by the PINNED scripts/issue509_scoring.py at 2d22b70c1 (extracted via git show at runtime, no branch switch); rebuilt raw rho_fe asserted to reproduce the stored sweep values exactly (rho_fe_reproduces_stored: true at both cells)
reconstruction	26 cells × 3 seeds (42 / 137 / 256): 18 cells from Claude Haiku 5-way judge labels (reanalysis_5way/judged_*_seed{42,137,256}.jsonl, HF rev 1b6e2053), 4 bare-Qwen-persona (qwen_default) cells from git-tracked judge aggregates, 4 Zelthari-scholar cells via substring proxy (SE/reliability path ONLY — point estimates stay the frozen CSV values); cross-check vs regression_data.csv passed, worst judge-cell abs diff = 0.0056 (1 judge flip / 180 completions)
reliability_y	panel 0.7785 (within-substrate pooled); per-cell on surviving pairs: anchor 0.787, search-best 0.821; Zelthari seed-256 calibration: substring proxy runs +0.080 to +0.087 above the judge across all 4 frames (consistent level offset, cross-frame spread 0.007)
inference	substrate-clustered bootstrap B = 5000 (seed 42), within-substrate permutation B = 2000, paired same-draw deltas vs each coarse predictor
headline (planned anchor, n = 25)	raw +0.364 → adjusted +0.411, adjusted CI [−0.010, +0.746], perm p = 0.085; paired lift vs coarse fact-slice JS on the common-finite pair subset = −0.333, CI [−0.457, 0.000]; pinned-convention point lift delta_rho = −0.223 (saturation-filtered) / −0.252 (unfiltered) — a different statistic on the pinned mixed-pair-set convention, carried without a CI
headline (search-best, n = 18)	raw −0.668 → adjusted −0.737, adjusted CI [−1.025, −0.444] (disattenuation treats reliability as known, so bounds can exceed 1 in magnitude — reported unclipped); double-controlled prior collapse −0.026, CI [−0.785, +0.714] (only 3 substrate clusters → at most 10 distinct cluster multisets; coarse quantiles, low power)
known artifact quirks (results.json)	n_seeds_in_se is a misnomer — it counts pairs with finite SE (25 / 18), not seeds; smoke mode's raw-correlation CIs use the pinned scorer's B = 5000 rather than the smoke B = 200 (production path unaffected); cluster counts are not stored in the JSON — the 3-cluster caveat is carried here and in the finding prose
inputs	per-seed judged completions + judge aggregates from the HF data repo at pinned revision 1b6e20530b1c6d477a387c18d5a88554910e7df9; target eval_results/issue_494/regression_data.csv; stored cell stats from eval_results/issue_509/fact_arm/scoring.json

Artifacts:

• Fact-arm scoring: eval_results/issue_509/fact_arm/scoring.json (953 scored cells + 560 N/A; smoke).
• Syco-arm scoring: eval_results/issue_509/syco_arm/scoring.json (1065 scored cells + 448 N/A; production stats with 5000-rep cluster bootstrap CIs clustered on source).
• Per-pair distance matrices: HF data repo superkaiba1/explore-persona-space-data issue_509/{fact,syco}_arm/bakeoff/metrics/ — 1513 JSON files per arm; 3026 metric files total across both arms.
• Bakeoff meta + truncation + cosine cross-check: HF issue_509/fact_arm/bakeoff/ and HF issue_509/syco_arm/bakeoff/.
• Parent leakage targets: fact arm reuses eval_results/issue_494/regression_data.csv; syco arm reuses eval_results/issue_480/_inputs/syco_411_analyze_summary.json (#411's frozen leakage matrix).
• Syco-arm judge artifacts: #411's raw completions (HF issue411_sycophancy_cosine_gradient/eval_results/), Claude Haiku per-source judge verdicts at eval_results/<source>/seed_42/judgments/ under the same tree (e.g. the assistant source's verdict files), judge-calibration artifacts (HF issue411_sycophancy_cosine_gradient/eval_results/judge_calibration/), and the frozen summary linked above.
• Per-seed judged completions consumed by the fact-arm production rerun: HF issue444_real_figure_provenance/.../reanalysis_5way/judged_*_seed{42,137,256}.jsonl and HF issue192_persona_spread/raw_completions/fact_seed{42,137,256}_e1*.json.
• Follow-up fact-arm production-rerun results: eval_results/issue_509/pathb-fact-rerun/results.json (2 headline cells with adjusted + double-controlled CIs and paired coarse deltas) and per_seed_reconstruction.json (26 cells × 3 seeds, per-cell scorer provenance, cross-check report); written before the figure.
• Follow-up fact-arm production-rerun figure: figures/issue_509/pathb_fact_rerun PNG + PDF + sidecar.
• Figure source PNG + PDF + sidecar: figures/issue_509/ — 3 figures (anchors_vs_502, layer_sweep, fact_prior_collapse), each with PNG + PDF + .meta.json sidecar.
• Follow-up base-rate-control results: eval_results/issue_509/baserate-partial-early-layer-cosine/results.json — 29 cells × 2 targets × 2 panels, per-source partials, degeneracy flag; on the issue-509 branch at the pinned commit.
• Follow-up figure: figures/issue_509/baserate_partial_early_layer PNG + PDF + sidecar.
• Follow-up bystander-bootstrap results: eval_results/issue_509/bystander-bootstrap-cis/results.json — 16 cells × 3 resampling axes, the paired-difference table, and pinned-input metadata; written before the figure.
• Follow-up bystander-bootstrap figures: two-panel summary (CI forest + paired differences) and the 8-cell forest variant PNG + PDF + sidecar.
• Parent task: #502 (28-layer bake-off, L22 gauss_kl winner). Fact priors: #494 (pooled-predictor null), #500 (prior-dominates verdict), #444 / #381 / #389 / #390 / #192 (adapter + matrix lineage). Sycophancy priors: #470 (cosine + JS null), #480 (marker null), #411 (adapters + 138-cell leakage panel). Recipe parent: #493 (8-layer / 50-probe bake-off), #474 (marker ΔG / G_logprob substrate).
• WandB project: n/a (predictor-only analysis, no training metrics to stream).

Compute: ~6.5 GPU-hours on 1× pod-509 (8× H100 80GB SXM5, terminated 2026-06-08T06:23:55Z after upload-verification PASS). CPU aggregation + scoring + figures on the dev VM, ~1 hour single-threaded. The base-rate follow-up is zero-GPU: ~2 min CPU on the dev VM. The bystander-bootstrap follow-up is likewise zero-GPU: ~2.5 min CPU on the dev VM. The fact-arm production rerun is likewise zero-GPU: ~35 s CPU on the dev VM.

Code:

• Driver: scripts/issue493_extraction_metric_bakeoff.py (inherited unchanged from #502 + surgical edits: --conditions-registry + --probe-pool-mode {q_test_strict, custom} flags; cond-ID regex [A-Z]\d+ accepts both FB1..FB9 and SC1..SC24 with no change).
• Dispatcher: scripts/issue502_dispatch.py (--phase metrics only; regress phase is bypassed because scoring is per-arm Spearman against per-arm targets, not the parent's ΔG).
• Scoring (NEW): scripts/issue509_scoring.py (~700 LOC including per-cell saturation screen, substrate-FE / source-FE residualization, double-FE prior-control, attenuation-adjustment via reliability_y, cluster bootstrap CI, within-stratum permutation null, delete-one-source jackknife on syco, comedian-recovery diagnostic, live-cells subset, coarse-predictor anchor; smoke vs prod path).
• Figures driver: scripts/issue509_figures.py — round-2 revision sources coarse anchors from each plotted cell's own per-coarse correlation field so all bars sit on the same n=18 cells.
• Follow-up base-rate-control script (NEW): scripts/issue509_baserate_covariate_earlylayer.py at commit fe8fdb308607d1c0801b9d106b0cd7f10f2ee630 on issue-509. Reproduce: uv run python scripts/issue509_baserate_covariate_earlylayer.py --bootstrap 5000 --perm 2000 (CPU-only, ~2 min; reads the pinned HF metric files + the frozen target matrix; writes results.json before the figure so a figure crash cannot lose the stats).
• Follow-up bystander-bootstrap script (NEW): scripts/issue509_bystander_bootstrap.py at commit 7f2869fa72d8bb9547747c90970cf177672cd142 on main. Reproduce: uv run python scripts/issue509_bystander_bootstrap.py --bootstrap 5000 --seed 42 (CPU-only, ~2.5 min; estimator helpers copied verbatim from the base-rate follow-up script with a hard cross-check against its stored geometry-alone values; reads the pinned HF metric files + the frozen target matrix; writes results.json before the figure).
• Follow-up fact-arm production-rerun script (NEW): scripts/issue509_pathb_fact_rerun.py at commit 258b0f60bdfc6955d0fdb3b8d44515697a12f161 on main. Reproduce: uv run python scripts/issue509_pathb_fact_rerun.py (CPU-only, ~35 s, deterministic: pinned HF revision, bootstrap seed 42; statistics delegated to the pinned issue509_scoring.py extracted via git show; fail-loud gates on the reconstruction cross-check, the stored-correlation reproduction assert, and input row counts; writes both JSONs before the figure).
• Fact-arm conditions registry: src/explore_persona_space/experiments/i509_fact_conditions.py.
• Syco-arm conditions registry: src/explore_persona_space/experiments/i509_syco_conditions.py.
• Git commits: worktree branch issue-509 at 2d22b70c1473786d17c6e9def0f28d744e4a119d for scoring + driver; main at 8d5c97451 for round-2 figures + body.
• Next steps (in priority order; three zero-GPU follow-ups were run on this issue and are folded into the findings above — the bystander-base-rate control, the bystander-resampled bootstrap CIs, and the fact-arm production rerun with per-seed reliability reconstruction): (1) prompt-text-similarity control — compute string-distance / token-overlap between the persona system prompts and test whether the early end-of-system cosine correlation collapses under that control, to rule out the "early-layer cosine reads system-prompt-string similarity" alternative — now the top remaining check on the sycophancy signal, since the base-rate control left it unattenuated (cost_class: free-analysis, headline_affecting: yes); (2) syco multi-seed target rerun — the #411 leakage matrix is single-seed-42 and the current syco result inherits that, so re-running #411 with two more seeds would close the seed-stability question on the L02-L08 cluster (cost_class: needs-gpu, headline_affecting: yes); (3) prior-only residualization diagnostic on fact — store ρ with bystander_logprob partialled out WITHOUT substrate-FE to disambiguate over-residualization from joint-FE collapse; the production rerun confirmed the joint-FE collapse with a CI but did not store the prior-only variant (cost_class: free-analysis, headline_affecting: yes); (4) held-out probe pool re-extraction — the syco result rests on the same 50 probes the bake-off optimized over, so re-running on a held-out probe pool would address the selection inflation across the ~1065 scored cells (cost_class: needs-gpu, headline_affecting: yes); (5) single-factor paired deconfound — extend the shared-draw paired-difference machinery to cell pairs differing in exactly one factor (metric family at fixed layer + extraction point; layer at fixed metric + extraction point) to attribute the global-max cell's edge to a factor, over the same stored matrices (cost_class: free-analysis, headline_affecting: no).
• One-block reproduce snippet (assumes a fresh 8× H100 pod with the repo cloned to /workspace/explore-persona-space, the .env set up, and the parent target matrices reachable):

cd /workspace/explore-persona-space && \
  nohup uv run python scripts/issue502_dispatch.py \
    --phase metrics --num-gpus 8 --batch-size 8 \
    --bakeoff-root eval_results/issue_509/fact_arm/bakeoff \
    --figures-root figures/issue_509/fact_arm \
    --probe-pool eval_results/issue_502/probes_500.json \
    --probe-pool-mode q_test_strict \
    --layers 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 \
    --transformations FB1 FB2 FB3 FB4 FB5 FB6 FB7 FB8 FB9 \
    --conditions-registry explore_persona_space.experiments.i509_fact_conditions \
    > /workspace/logs/issue509_fact.log 2>&1 && \
  nohup uv run python scripts/issue502_dispatch.py \
    --phase metrics --num-gpus 8 --batch-size 8 \
    --bakeoff-root eval_results/issue_509/syco_arm/bakeoff \
    --figures-root figures/issue_509/syco_arm \
    --probe-pool data/issue_509/eval_50_probes_for_extraction.json \
    --probe-pool-mode custom \
    --layers 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 \
    --transformations SC1 SC2 SC3 SC4 SC5 SC6 SC7 SC8 SC9 SC10 SC11 SC12 SC13 SC14 SC15 SC16 SC17 SC18 SC19 SC20 SC21 SC22 SC23 SC24 \
    --conditions-registry explore_persona_space.experiments.i509_syco_conditions \
    > /workspace/logs/issue509_syco.log 2>&1 && \
  uv run python scripts/issue509_scoring.py --arm fact --smoke && \
  uv run python scripts/issue509_scoring.py --arm syco && \
  uv run python scripts/issue509_figures.py