The marker's prior-on-LEVEL / geometry-on-CHANGE split does not cleanly transfer to sycophancy: prior and geometry are tied on the absolute level on both arms, and both arms' trained shift is below the registered precision gate (MODERATE confidence)

Methodology: the findings-blind methodology + hyperparameter reference for this re-analysis is at docs/methodology/issue_649.md @ 6a94a0d · gist.

Takeaways

• On the absolute trained level (LEVEL), base prior and cosine geometry tie on both arms (on-policy +0.55 each over source intercepts). The marker's "prior dominates LEVEL" rule does not transfer.
• The earlier "+0.55 prior vs +0.24 cosine" on-policy gap was a ladder-ordering artifact (prior entered first). Read symmetrically, cosine matches prior (marginal +0.82 vs +0.77, overlapping CIs).
• Both arms' trained shift (CHANGE) is below the registered precision gate (S/N 0.89 canned, 0.18 on-policy). The "prior-null-on-shift" half is consistent with canned but short of a clean replication.
• Cosine predicts CHANGE (canned pooled +0.57); Gaussian-KL does not (null at L2, sign-reverses at L7). The geometry→CHANGE claim is cosine-specific, not geometry-general.
• Cosine↔CHANGE is not source-uniform (villain +0.77 to software-engineer −0.12) and holds at L2/L7/L20. Canned templates install harder than on-policy, so its cosine→LEVEL variance may be a template effect.

What I ran

• Why: The parent marker experiment (#532) found a clean two-component rule: a bystander's base prior predicts the absolute trained leakage level, activation geometry predicts the trained-minus-base shift, and prior is null on the shift. Two sycophancy siblings disagree on whether prior or geometry wins the single-DV race (#507 prior wins at 72B; #509 geometry wins at 7B), and the fact-leakage line found base prior a durable level predictor (#500, #541). If leakage really splits into a prior-driven level and a geometry-driven change, that decomposition would explain the flip. This task tests whether the marker's split transfers to a realistic judged behavior, and whether it depends on how the behavior was installed (#612's canned-vs-on-policy contrast, which #627 showed matters for install strength).
• Design: Pure CPU re-analysis of an existing sycophancy panel. No new training. Per (source × bystander) cell, leakage splits into LEVEL (absolute trained agreement rate) and CHANGE (trained − base rate); two predictors race on each DV: the bystander's base agreement prior, and source→bystander early-layer activation geometry (centered cosine + Gaussian-KL). Two arms: canned-template positives (4 sources × 30 bystanders, 108 cells, coverage anchor) and on-policy positives (2 sources, 55 cells, realism confirmation), two seeds averaged per cell.
• Training: N/A. No training in this task. The trained agreement rates are inherited from the substrate panel's already-trained adapters.
• Eval: DV inputs are the inherited on-policy Haiku-judged agreement rates (60-claim held-out wrong-claim pool, 600 verdicts per cell). Predictor inputs are base Qwen-2.5-7B-Instruct residual-stream activations over the 34-persona system-prompt set, re-extracted at the early-layer band where sycophancy geometry was previously shown to live (end-of-system layer 2 cosine, primary; last-prompt layer 7, secondary; deeper layer 20, robustness). Verdict per DV: six-regression incremental-validity CV-R² ladder (bystander-grouped 5-fold CV) plus marginal Spearman with 1000-bootstrap 95% CIs. Goal was refined once during planning (see events.jsonl).

Findings

On the absolute level (LEVEL), prior and cosine are tied on both arms; the marker's "prior dominates LEVEL" rule does not hold

The marker rule needs base prior to dominate the trained rate. The honest test enters each predictor symmetrically, taking its CV-R² uplift over the source-intercepts model, so add-order does not pick the winner.

Figure. Entered apples-to-apples over the source intercepts, prior and cosine are tied on LEVEL on both arms (canned prior +0.19 / cosine +0.26; on-policy +0.55 / +0.55). Held-out CV-R², bystander-grouped 5-fold CV; canned 108 cells, on-policy 55 cells.

• On-policy (right): cosine over intercepts +0.55, prior +0.55, a dead tie. Marginal cosine↔LEVEL (ρ +0.82, CI excludes 0) matches prior (+0.77, CI excludes 0), overlapping CIs. The earlier "+0.55 prior dwarfs +0.24 cosine" gap was a ladder-ordering artifact (prior entered first). LEVEL spans 0.035 to 0.812, far above the Wilson floor, so a floor effect is excluded.
• Canned (left): cosine +0.26, prior +0.19; marginals tied (+0.51 each). Tilts slightly to geometry, statistically indistinguishable.
• Geometry is at least as strong as prior on LEVEL on both arms; the marker's prior-owns-LEVEL half does not transfer, and the result is symmetric across arms.

Both arms' trained shift (CHANGE) is below the registered precision gate; the "prior-null-on-shift" half stays a coarse rate-space read, short of a clean replication

The marker's signature was: base prior tracks the level but says nothing about the training-induced shift. The right-hand panels show that canned pattern, but the precision gate fired here too.

Figure. In rate space, base prior predicts the absolute level but is flat on the shift; early-layer cosine predicts both. Each point is one of 108 source×bystander cells (canned arm). Titles carry the marginal Spearman ρ with bootstrap 95% CI. The CHANGE column is below the registered precision gate (S/N 0.89).

• Prior↔LEVEL is positive (ρ +0.51, CI excludes 0); prior↔CHANGE is flat (ρ +0.14, CI covers 0), matching the marker's prior-null-on-shift signature. The registered precision gate fired on canned (S/N 0.89, under its threshold of 2); the planned forced-choice probe was not run. Consistent with the signature; short of a confirmed replication.
• On-policy CHANGE is further below floor (S/N 0.18). Its prior↔CHANGE marginal does exclude zero (ρ +0.58, CI excludes 0), so I do not call it "prior-null"; at this S/N the non-null read is uninterpretable.
• The floor caps the shift while leaving the directly-measured level unaffected, which is why LEVEL stands on both arms while CHANGE does on neither. Collinearity is ruled out (Pearson |cosine|↔prior −0.03 / +0.05, far below the 0.6 gate).

Cosine carries the rate-space CHANGE pattern, but only cosine, and not uniformly across sources

"Geometry predicts CHANGE" is really "cosine geometry predicts CHANGE": the other registered geometry predictor, Gaussian-KL, is null at L2 (canned KL_L2↔CHANGE ρ −0.13, CI covers 0) and reverses sign at L7 (−0.61). The pooled cosine↔CHANGE = +0.57 also hides per-source heterogeneity.

Figure. Three of four sources show a strong positive cosine→CHANGE slope (villain +0.77, comedian +0.68, kindergarten-teacher +0.63); the software-engineer source reverses sign (−0.12, p = 0.55). Per-source Spearman on the canned arm, n ≈ 27 cells each.

• Three sources are strongly positive (p ≤ 0.001); software-engineer reverses to ρ = −0.12 (p = 0.55). The cosine→CHANGE result is a bystander-grouped aggregate read; it does not hold uniformly when split by source.
• Gaussian-KL does not carry the pattern at all, so the geometry→CHANGE claim is specific to cosine, not to "activation geometry" broadly.

Cosine→CHANGE is not early-layer-specific: it holds at the early, last-prompt, and deeper layers alike

The cell was first identified at the early layer (L2), but the registered design also extracted last-prompt (L7) and deeper (L20) cosine. If the effect were an early-layer accident, it should fade with depth.

Figure. Cosine→CHANGE holds at the early (L2), last-prompt (L7), and deeper (L20) layers on both arms; every CI excludes 0. On-policy L20 is strongest (+0.81). Bystander-grouped, 1000-bootstrap CIs.

• Cosine predicts CHANGE at every layer (canned +0.57 / +0.57 / +0.34, CI excludes 0, at L2/L7/L20; on-policy L20 strongest +0.81). The "early-layer" label simply marks where the cell was first identified, not where the effect lives.
• Competing read for canned LEVEL: canned positives are fixed templates that over-install (+0.84–0.93 vs on-policy +0.60–0.66, per the matched-install reference), so the cosine→LEVEL variance may be a template-installation artifact. Consistent with the symmetric on-policy LEVEL tie, where the artifact is absent.

Data

Trained on

N/A — no training in this task. The trained agreement rates are inherited from the substrate sycophancy panel's adapters (4 source personas trained to agree with false user claims; canned-template and on-policy positive arms, contrastive negatives, two seeds). The complete training mix and judgments live on the HF data repo: superkaiba1/explore-persona-space-data @ 14d541b.

Evaluated with

The dependent variables are the inherited on-policy agreement rates: each cell is a source persona's adapter evaluated on a bystander persona over a 60-claim audited held-out wrong-claim pool, 10 rollouts × 60 claims = 600 free-generation verdicts, judged by Haiku (κ = 0.869 vs Sonnet). LEVEL = trained agreement rate t; CHANGE = t − b where b is the bystander's base agreement rate (52 personas measured pre-training). Predictors: base prior b; centered cosine and Gaussian-KL (16-D subspace, ≥32 probes/persona) between source and bystander residual activations at the early-layer band. Cells excluded: the diagonal (source = bystander) and any bystander that is a trained contrastive negative for that source (the disjointness invariant). Realized: 108 canned cells, 55 on-policy cells.

3 of 108 canned cells, random sample (seed 42), shown seed-averaged (LEVEL = mean of the two trained seeds t_seed_42/t_seed_137, the analysis DV — not the seed-42-only rate):

source             bystander            LEVEL(t)  CHANGE(t-b)  prior(b)  cosL2
software_engineer  villain              0.375     +0.322       0.053     -0.206
villain            pirate_captain       0.307     +0.219       0.088     +0.051
comedian           web_developer        0.006     -0.036       0.042     +0.047

Full per-cell table (both arms, all predictors, per-seed t_seed_42 / t_seed_137 columns): eval_results/issue_649/per_cell_table.csv @ e454897. The complete base + trained judgment files (each with raw verdicts): HF data repo @ 14d541b.

Generated

N/A — this task generated no model completions. The model generations underlying the DV are the inherited on-policy rollouts, in the raw-completions tree linked above.

Reproducibility

• Methodology reference: docs/methodology/issue_649.md @ 6a94a0d · gist — findings-blind methodology + hyperparameter table.

• Parameters:

  Parameter	Value
  Task type	CPU re-analysis (no training, no pod)
  Substrate	#612 sycophancy panel (4 sources, 30-persona graded-cosine panel)
  DVs	LEVEL = trained rate; CHANGE = trained − base rate
  Predictors	base prior; centered cosine + Gaussian-KL @ early-layer band
  Geometry layers	end-of-system L2 (primary), last-prompt L7 (secondary), L20 (robustness)
  Gaussian-KL subspace	k = 16, ≥32 probes/persona
  CV	bystander-grouped 5-fold (headline); source-grouped + intercept-only M0 (robustness)
  Bootstrap	1000 reps, Spearman 95% CI
  Collinearity gate	Pearson(|cosine|, prior); canned −0.03, on-policy +0.05 (both PASS)
  Precision gate (#391)	FIRED both arms (S/N canned 0.89, on-policy 0.18) — caps CHANGE only, not LEVEL
  Cells (after exclusions)	canned 108, on-policy 55
  Seeds	42, 137 (averaged per cell)
  Config slugs	arm_canned, arm_onpolicy; ladder models M0–M5

• Artifacts:

  • CV-R² ladder: eval_results/issue_649/cv_r2_ladder.json @ e454897
  • Marginal Spearman + non-circular partials: marginal_spearman.json @ e454897
  • Precision + collinearity gates: precision_check.json, collinearity_gate.json
  • Per-cell table: per_cell_table.csv @ e454897
  • Figures: figures/issue_649/ @ e454897

• Compute: 0 GPU-hours (CPU re-analysis on the VM; geometry re-extraction ran CPU-only over 34 short prompts). No pod provisioned.

• Code: extractor scripts/issue649_extract_panel_earlylayer.py; analysis scripts/issue649_level_change_decomp.py; hero re-plot scripts/issue649_hero_replot.py; git commit 904c2668 (analysis run), e454897 (round-3 figures). Reproduce: uv run python scripts/issue649_level_change_decomp.py then uv run python scripts/issue649_hero_replot.py.

• Reused artifacts:

  • Reused sycophancy panel + judgments from #612: HF data repo @ 14d541b — fit: same base model (Qwen-2.5-7B-Instruct), on-policy judge-scored rates over the audited held-out claim pool, all 4 sources × 30 bystanders + base rates present; the realism regime is exactly the one this decomposition reads off.
  • Reused base agreement rates from #612: eval_results/issue_612/base/judgments/ (52 personas) — fit: the bystander base prior b, measured pre-training on the same probe pool.

• Context:

  • Created / run: created 2026-06-15; analysis + figures landed 2026-06-16.
  • Follow-up to: #532 — the marker LEVEL/CHANGE decomposition this task ports to a realistic behavior.
  • Originating prompt(s), verbatim:

    Can we run a followup to check effect of base prior on change in leakage for the more realistic behaviors?