Scaling context-to-answer maps narrows probe deficits but does not establish unlabeled pairs as a substitute for judged pairs (HIGH confidence)

Methodology: updated standalone methodology · gist mirror

Latest follow-up, 2026-09-18: completed the natural-prompt high/low extraction comparison, preserving all original evaluation rows and layers. Natural mapped OOD scores are 0.172/0.123/0.145 for sycophancy/hallucination/evil versus E1 0.210/0.215/0.209. Mapping helps the primary natural sycophancy/evil directions; tail sensitivity and weak evil elicitation limit the claim. Full report.

Earlier follow-up, 2026-09-16: the fixed-answer-direction comparison is complete using the 963,444-pair generic map, without downstream behavior regression. Positive transfer is dataset dependent and does not consistently beat the context-native direction. Five natural-dataset cells are informative; three WildChat cells retain only four contexts each. Full methods, paired results, and limitations are appended under “Fixed answer-direction transfer — 2026-09-16.”

Earlier follow-up, 2026-09-08: the requested no-recombination sweep is complete at 150/150 cells, through 100,000 generic pairs plus fixed trait pools. The new P-B result is appended at the end under “Natural context–answer scaling follow-up”: a hallucination advantage (+0.0528 mean ρ at 100k), with slightly negative mean gaps for evil and sycophancy. The historical title, Goal, and prior-round classifications below are preserved; they are not a classification of this new descriptive follow-up.

Takeaways

• A seven-rung, five-seed unlabeled-data ladder improves the mapped-answer-minus-context-probe difference in 8 of 9 primary panels, but the full-store difference remains negative in 8 of 9: 0 panels are supported, 8 refuted, and evil generic transfer is unresolved.
• Projecting persona vectors on the linearly mapped answer performs near the real answer (evil in-distribution 0.671 vs 0.688; sycophancy OOD 0.240 vs 0.250) and far above context projection off-synthetic (0.671 vs 0.031); exceptions: hallucination OOD keeps a real oracle gap (0.215 vs 0.353) and sycophancy-synthetic needs the MLP map (0.701 vs 0.764).
• Direct linear probes beat every persona-vector method off-synthetic (sycophancy OOD 0.71-0.73 vs 0.24-0.25).
• Across 13 fair-allocation rungs, mapped-answer leads on 10 and clears shuffle on 12 (both medians +0.03); only sycophancy's flagship clears both interval criteria. The repaired context-direction comparator provisionally supports the edge for sycophancy and hallucination (median +0.12 over the 8 sanity-passing rungs; the admitting sanity gate is itself unhardened); evil's comparator still fails its in-distribution band and stays indeterminate.
• The original grid's adverse aggregate stands for its configuration: with a fully generic map pool at large labeled budgets, map-then-project trails the context-extracted direction (below zero in 712/810 evil cells), and on the fully held-out attack-tactic family map arms do not separate from a shuffled-direction control (holdout under-determined: 1,906 train rows vs 3,584 dimensions).
• Flip contrast +0.38 for evil (interval excluding zero; 20 of 20 seed-anchor terms positive) versus +0.04 sycophancy and -0.02 hallucination (intervals include zero): the in-domain map-pool flip is evil-specific.

Goal

This experiment in context: Persona-vector monitoring projects an answer-derived behavior direction onto context activations — a datatype mismatch this experiment tries to correct by applying the learned context-to-answer map first, scoring each context by projecting the mapped activation instead of the raw one. It reuses the #1092 WildChat+LMSYS activation store as the unlabeled map corpus, the #779 direction-extraction recipe, and the #722/#779 batched map-fitting stack. The primary comparison fixed in the plan: map-then-project versus the context-native direction at matched unlabeled and labeled budgets, across three behaviors and a real-data distribution-shift ladder.

Broader narrative: If a map learned from cheap unlabeled text closed the context/answer datatype gap, behavior monitors could reach a target accuracy with far fewer behavior labels and degrade more gracefully off-distribution. The measured answer is that on generic-corpus maps it does not — direct regression on context activations stays the default monitoring predictor, and map benefits from in-domain map pools are confirmed only for evil — indeterminate for sycophancy and hallucination.

Methodology

Design: Prediction experiment on Qwen2.5-7B-Instruct — no fine-tuning; all arms are read-outs over frozen activations. 16 predictor arms (5 context-side, 5 map-based, 2 true-answer oracles, 4 controls) score every context; the target is on-policy behavior expression. Grid per behavior: 2 mapping variants (prefix-based and context-based, both run per the standing rule) x direction regimes (E1 = synthetic contrastive system-prompt pairs per the persona-vectors recipe; E2 = matched natural pairs; E2p = pooled natural — concrete constructions under Data extraction; hallucination E1 only) x unlabeled map budget U in {250; 5,000; 18,793 = full store} x labeled budget L in {250; 2,500; 8,000 for evil / 16,000 otherwise} x 5 label draws x 3 seeds — 810 evil grid cells + 16 evil composition cells, 810 sycophancy, 270 hallucination (1,906 total), plus 90 reversed-transfer evil cells (train on hh-rlhf red-team, evaluate on the held-out jailbreak slice + ToxicChat). Per cell: group-level 5-fold round-robin cross-validation (conversation / persona / jailbreak-family groups), all 28 layers swept, frozen-best-layer rho with selection-inherited paired-bootstrap intervals (500 draws) plus nested layer selection, and a max-over-arms-and-layers permutation null (500 draws). Follow-up round new-arm-round (run 2026-08-02/03, twelve single-A100-80 boxes) adds, with every cross-round join made at the full unlabeled pool only: (1) a final-context extraction point for the behavior direction, re-running the three direction-dependent projection arms for both extraction points; (2) nonlinear true-answer oracles (MLP and kernel ridge) beside a within-round linear-oracle rider at identical coordinates — the rider reproduces the committed linear-oracle rows exactly (210 matched full-pool train coordinates joined on behavior, budget, draw, seed, regime, and variant; largest absolute difference zero) — the hallucination nonlinear leg ran only the 250- and 2,500-label budgets, while the committed hallucination linear-oracle grid reaches 16,000 labels at the joinable full-pool coordinate (the round summary's u=250 note describes only its transfer-rung rows); (3) the previously missing transfer-rung roster — direct MLP and the MLP/kernel-map ridge readouts. The round merged 18,054 transfer rows and 7,974 per-cell rows with zero duplicate drops, zero recorded skips, and zero roster-accounting violations. Follow-up round evil-ood-spread-round (run 2026-08-03/05) attacks the standing evil transfer-rung spread failure with four items over the existing evil rollouts plus three new pilot corpora: (A) the two-sided DV spread gate on three fresh attack-family corpora, 200 contexts x 5 rollouts each, near-duplicate-screened against the train scrape; (B) an attack-tactic-family holdout inside the jailbreak train rung — every train context judge-labeled into a seven-class attack-tactic taxonomy, the largest class (Direct Request, 4,562 of 6,468 contexts) held out entirely, all 16 arms refit on the remaining 1,906 contexts and scored on the holdout, both mapping variants and all three direction regimes, 500-draw bootstrap intervals; (C) a planned detection-metric re-cut, dropped as superseded by a concurrent follow-up round (no committed artifact); and (D) a graded 0-100 compliance DV judged over all 159,990 retained evil rollout draws on the three existing rungs. Every spread verdict in this round uses one instrument on both sides of the comparison: per-context mean scores, sample SD, strict bottom bin. The fair-allocation readout protocol (slug P-B, banked fair-refit scoring rounds of 2026-08-06, rows at code 5aae0a472b) trains every label-consuming arm on one shared pool — an 80% group-level slice of every trait-eliciting dataset except one dataset held out whole, plus the judged WildChat train split — and scores it on the whole held-out rung, so probe and projection arms see identical labeled budgets; its rung grid spans 13 whole-holdout rungs: evil's two original transfer rungs plus full-corpus versions of the three attack-family corpora piloted above, sycophancy's held-out Reddit slice plus five real OOD rungs (Data extraction), and hallucination's two transfer rungs. (The restrictive single-dataset pool variant, slug P-A, trains on one trait-eliciting dataset plus the WildChat split.) Follow-up round claim4-controls (run 2026-08-19; four production RunPod CPU pods plus three OOM-superseded ones) adds four controls on that protocol's mapped-answer-vs-context probe comparison: (1) a shuffled-pairing map control — the context-to-answer map refit after one within-component permutation of the answer side of its fit pool (generic and trait-eliciting blocks permuted separately, one permutation shared across all layers; whitening, readout pools, eval rungs, and frozen layers held identical to the true-map pass); (2) 5-seed replication (seeds 0-4) of both map variants with per-rung seed t-intervals and paired group-level context bootstraps; (3) the context-extracted direction run as a transfer-mode comparator arm (new this round), beside a weight-permutation map control; (4) the corrected sycophancy OOD table surfaced from banked single-seed rows with per-row provenance labels. The round generates zero new rollouts and zero judge calls, and a seed-0 reproduction gate reproduced the banked rows with zero drift at the 1e-9 report tolerance (rc=0 on all three behaviors) before the remaining seeds ran. Follow-up round composition-grid-multiseed-plus-arm2-repair (run 2026-08-23/24, ten verified RunPod CPU pod chains; zero new rollouts, judge calls, or labels) adds two legs.

Leg 1 re-runs the composition sub-grid at 5 seeds x 3 behaviors on one fixed 5,000-row unjudged map pool whose in-domain share f_u varies over 0 to 1 (anchor cells at shares 0 and 0.5 x labeled budgets 250 / 2,500 / 8,000; dose rungs at 2,500 labels; a companion flag f_l sets whether the in-domain half may include the anchor's own labeled contexts, with f_l = 0 the held-out reading) — realized 110 evil + 130 sycophancy + 130 hallucination cells (evil's 20 designed skips are recorded infeasible-pool recipes: the fully in-domain rungs and the no-overlap 8,000-label anchors exceed its 6,468-context pool). The verdict statistic fixed in the plan is the flip contrast C — per seed, the mean over anchor cells (both variants, budgets 250 and 2,500, held-out pools) of the change in the map-minus-context difference from share 0 to share 0.5 — confirmed only when the 5-seed t-interval excludes zero. Sycophancy's prefix variant (65 realized cells) is excluded as designed-degenerate — its contexts carry no system prompt, so the prefix substrate is constant by construction — and its criterion reads over the 2 remaining anchors, a stated deviation from the 4-anchor grid. Execution notes: each behavior ran as two seed-halves (0-2 and 3-4); the sycophancy seeds-3-4 half split into single-seed pods after repeated first-cell wall halts on slow hosts, and the sycophancy seeds-0-2 chain's eigendecomposition crash was fixed by a jittered solver with a singular-value-decomposition terminal fallback, its fix-engaged signal confirmed in the production log before results were accepted.

Leg 2 repairs the claim4-controls context-direction comparator — specifically its transfer adapter, the construction that turns the committed context-extracted direction into a transfer-mode scorer (a closed-form direction fit, not a LoRA). Sycophancy and hallucination keep v1, the committed comparator construction unchanged with the repair confined to the transfer wiring (275 and 55 realized transfer rows). Evil takes the quantile-restricted composite (v2) — the direction refit on the top and bottom quartiles of the pool behavior score, a separately named estimator — with parity refits: the mapped-answer arm refit on the identical restricted rows so both sides see the same label budget (610 rows). A pre-fixed sanity gate admits a behavior only when its comparator's in-distribution read lands inside the banked 45-cell band; the mapped-answer-minus-comparator difference is read only over sanity-passing behaviors.

The unlabeled-data-ladder follow-up (run 2026-09-05) isolates generic unjudged-pair volume without generating new rollouts, judge calls, or labels. For each behavior and each of five seeds it fits true-pair and within-component pairing-shuffled maps at U in {250, 500, 1,000, 2,000, 5,000, 10,000, 18,793}, using full-union whitening, then compares arm 7 (mapped-answer ridge) with arm 4 (context ridge) and arm 12 (true-answer oracle) in in-distribution, generic-transfer, and OOD setting groups. The primary read uses generic-only map pairs; union-scaled maps are secondary. The statistic is D(U) = rho(arm 7) - rho(arm 4). A ladder is supported only when the five-seed trend's 95% interval is above zero, the full-U endpoint is not equivalent to zero, and the judged-minus-generic contrast is equivalent to zero. “Equivalent” was predeclared as a two-one-sided test at alpha 0.05: the entire 90% interval must lie inside (-0.02, +0.02), meaning any remaining advantage or deficit is practically smaller than 0.02 rho. A ladder is refuted when the endpoint is equivalent to zero or the judged-minus-generic contrast's 90% lower bound is at least +0.02; all other outcomes are unresolved.

Training: N/A — no model training. Evaluation/generation hyperparameters:

Hyperparameter	Value	Source
Base model	Qwen/Qwen2.5-7B-Instruct	constants.py MODEL_NAME @ eb084ff2c4
Rollouts per context (K)	5	constants.py K_ROLLOUTS
Generation temperature / max new tokens	1.0 / 1,024	generation.py GEN_TEMPERATURE, GEN_MAX_NEW_TOKENS
Judge model	claude-sonnet-4-5-20250929	constants.py JUDGE_MODEL (project pin)
Judge draws per completion / temperature	3 / 1.0	constants.py N_JUDGE_DRAWS, JUDGE_TEMPERATURE
Judge max tokens	400; 800 re-judge for truncation-affected items	constants.py; judge_summary.json rejudge_800
Ridge lambda grid (map + regression arms)	0.01 to 1,000 (6 log steps)	constants.py RIDGE_LAMBDAS (store-producing-run lineage)
MLP arms	width 512, one hidden layer, max 300 epochs	constants.py MLP_HIDDEN, MLP_MAX_EPOCHS
Folds / bootstrap draws / permutation draws	5 group-level / 500 / 500	constants.py N_FOLDS, N_BOOT, N_PERM
Direction extraction (E1)	5 pos/neg pairs x 20 questions x 10 rollouts, judge-filtered	plan §0 (arXiv 2507.21509 recipe)
Activation store	reused WildChat+LMSYS store, rev e5901706, fp16, 28 layers x 3,584	constants.py STORE_REVISION
Final-context direction (round 2)	diff-of-means at the final-context token, same judge-filtered rows as the answer-averaged direction (position is the only change)	plan v9 §11 — deliberate deviation from the arXiv 2507.21509 response-average step
Oracle MLP (round 2)	direct-MLP recipe: width 512, one hidden layer, 300-epoch AdamW multihead, input = whitened true answer activation	direct-MLP block in experiments/issue_1739/arms.py @ a78f971b51
Oracle kernel ridge (round 2)	median-heuristic gamma, inner-validation (gamma, lambda) selection	plan v9 §11 (nonlinear-map round recipe)
Round-2 replicates	linear/oracle legs 5 draws x 3 seeds; nonlinear-map legs 3 draws x 2 seeds (reported per family, never pooled)	plan v9 §5
Compliance DV judge (evil-ood-spread round)	0-100 compliance rubric (reason-then-JSON), 3 draws, temp 1.0, max tokens 1,024, Batch API	compliance_dv_results.json (rubric_family, n_draws, max_tokens)
Attack-family pilots (evil-ood-spread round)	3 corpora x 200 contexts x 5 rollouts; trait judge, 3 draws, max tokens 1,024, sync route	pilot_spread.json (route, n_draws, max_tokens)
Tactic holdout (evil-ood-spread round)	16 arms, 1,906 train / 4,562 held-out contexts, 500 bootstrap + 500 permutation draws	item_b/holdout_metrics.json meta
Spread floor (all rounds)	SD at least 10 of 100; under 80% of contexts in the bottom (score under 10, strict) and top (90 or above) bins; per-context means, sample SD	gates.py spread-gate constants; plan §7
Claim4-controls grid	13 rungs x 5 seeds x two map variants (true, pairing-shuffled), fair-allocation protocol only	plan v21 §3/§4
Pairing shuffle (claim4-controls)	one within-component permutation of the answer rows (generic and eliciting blocks separate), shared across all 28 layers, rng [1739, 21, seed]	issue1739_r2v2_score.py @ 067bd0300c
Claim4 intervals	seed t-interval (5 seeds, df 4) + paired group-level context bootstrap, 2,000 draws	claim4_per_rung_table.json meta.n_boot
Claim4 reproduction gate	seed-0 true-map rows joined to banked rows at 5aae0a472b; report tolerance 1e-9, halt on absolute rho drift over 1e-3	issue1739_arm12_repro_check.py claim4 mode
Compose multiseed grid	5 seeds x 1 draw; fixed 5,000-row pool; in-domain shares 0 / 0.1 / 0.25 / 0.5 / 0.75 / 1; labels 250 / 2,500 / 8,000	compose_ms_table.json provenance @ 25af31bd3a
Flip-contrast verdict rule	C over anchors (2 variants x labels 250 and 2,500, held-out pools); confirmed iff the 5-seed t-interval excludes 0; banked seed-0 repro tolerance 0.005	issue1739_compose_ms_fold.py @ a0f14160d1
arm2fix sanity gate + intervals	in-distribution comparator read inside the banked 45-cell band; 5-seed t-interval + 2,000-draw paired context bootstrap	arm2fix_table.json sanity/meta @ f0530e0e8b
Unlabeled-data ladder	U = 250 / 500 / 1,000 / 2,000 / 5,000 / 10,000 / 18,793; 5 seeds; equivalence margin 0.02; TOST alpha 0.05; 95% trend and 90% equivalence intervals	eval_results/issue_1739/uladder_fold/uladder_fold.json @ 7202c3671c0

Evaluation: Primary DV per context = mean over K=5 on-policy rollouts of graded judge scores: evil and sycophancy use a 0-100 trait rubric (3 judge draws per completion, mean-aggregated; malformed/refusal returns dropped, never coerced), hallucination uses the fraction of rollouts judged fabricated under a three-way correct/abstained/fabricated rubric. Judge content-drops: 46,602 evil draws (29% — refusals on jailbreak-completion content; 1,811 of 10,666 evil contexts lost every draw and are excluded), 5,859 sycophancy draws (2.3%); zero transport losses; a truncation re-judge at 800 max tokens recovered 356 evil and 652 sycophancy all-dropped items. Predictor validity = Spearman rho between the arm's predicted score and the DV on held-out group folds; the headline read is map-then-project minus context-native with paired-bootstrap intervals. Undefined rank correlations (tie-degenerate projections) are recorded as missing, never coerced, and the singular-slice degenerate-fit fallback fired in zero of 1,906 cells. The spread floor (SD at least 10 of 100 and under 80% of contexts in the bottom bin) passes on every train rung (SD 26.3 evil, 13.2 sycophancy, 40.5 hallucination rescaled) and on the hallucination/sycophancy eval rungs; it fails on both evil transfer rungs (hh-rlhf red-team SD 0.9; ToxicChat 93% bottom bin) — round-2 reads exclude those two rungs from every hypothesis read (no zero bars, never silently averaged). The evil-ood-spread round re-measured that failure with a matched instrument on both sides of every comparison (per-context means, sample SD, strict bottom bin; results below) and adds two records: compliance-wave drop accounting is 2,086 of 159,990 draws content-dropped (1.30%) with zero transport losses, and a judge-route confound stays open — the pilot corpora were judged over the sync API route while the parent trait pools and the compliance wave were Batch-drained, and the Batch drain's parse loss is corpus-dependent (measured 2.7-64% vs 1.8-2.8% sync on the pilot corpora at a fixed instrument), so cross-route level comparisons are suspect until the library fix lands (routed to an infra task). The planned teacher-forced fixed-pool margin companion DV was not computed, so the graded rate carries all results alone; item-aligned split-half ceilings are 0.89 (evil) and 0.88 (sycophancy), none was computed for hallucination (values in intrusion_audit/recount.json). Round 2 adds no new rollouts, judge calls, or labels — every round-2 arm re-fits over the existing labels and activations. The claim4-controls round likewise consumes banked DVs only; its reads are paired per (behavior, rung, seed): the true-map delta (mapped-answer probe minus context probe, Spearman rho at the frozen layer), the same delta under the pairing-shuffled map, and their difference (the mechanism margin), each carrying a seed t-interval and a paired group-level context bootstrap (2,000 draws), reported separately and never conflated. The plan's verdict criterion requires positive medians over the 13 rungs plus flagship intervals clear of zero; the two evil rungs failing the parent trait-DV spread floor stay in the plan's 13-rung denominator, so reads there inherit that caveat. The shuffle's manipulation check is structural (a verified within-component bijection) with retrieval as an advisory read; the evil ToxicChat graded-compliance companion was a declared skip — per-context prediction-to-DV join coverage measured 77.3%, below the 90% join-coverage gate the plan fixed — never a partial silently scored. Four open review concerns from this round ride as deferred hardening items, none touching the folded numbers: claim4-seed-rerun-idempotency — residual seed-re-run resume windows (a stale same-commit sentinel; input-sha verification ordered after the sentinel write); claim4-lattice-rung-identity — a fold-script roster-note skip on an empty realized rung set (diagnostic layer only; this round realized 13 of 13 rungs with zero gaps); claim4-pushdown-production-bitidentity — single-chunk-only exact-agreement tests for the out-of-memory row-index pushdown (its outputs matched at full production shape under the seed-0 reproduction gate, rc=0 on all three behaviors); and claim4-chunk-turnover-rss — a chunk-turnover memory peak nearer 92-95 GiB than the in-code estimate (mooted operationally by the 256 GB boxes). The composition-multiseed round likewise consumes banked labels and activations only; each compose cell records the standing mapping-baseline reads beside the arm scores (held-out map R-squared up to 0.495 against an identity-plus-learned-bias baseline that never exceeds -0.001; top-1 retrieval medians up to 0.61) plus the anchor-overlap sensitivity recompute (largest shift 0.007, no verdict flipped), and the arm2fix leg's coverage checks report 8 of 8 planned rungs and a 40-of-40 join on behavior, fit, rung, and seed. Seven further open hardening concerns from this round's reviews ride as verification-hardening items (adjudicated concern-grade at the round's binding cap review; none touches a folded number): arm2fix-parity-currency-fail-open, arm2fix-resume-config-gap, and arm2fix-context-bootstrap-gap are ledger-deferred (per-id markers under Takeaways); arm2fix-d0-phase-idempotency, arm2fix-midpoint-helper-drift, arm2fix-legacy-byte-identity, and arm2fix-pvsynth-fit-axis-self-derived remain open. Two more stay open as interpretation bounds rather than deferrals — arm2fix-sanity-band-budget-universe-self-derived (BLOCKER-graded in the ledger, adjudicated concern-grade by the binding cap-round reconciliation, still open: the sanity band's budget universe is self-derived from the artifact rather than fixed in advance per behavior) and cms-smoke-phase-coverage (staging, halt monitoring, upload, and a tiny-real fold sat outside the round's certified smoke surface) — they bound the repaired-comparator read, reported as provisional in Results.

Data extraction: The map M is a ridge fit from store context_end (or prefix_end) activations to answer-span mean activations, fit on U rows of the reused WildChat+LMSYS activation store (18,793 rows after excluding the 2,400 eval-only battery rows; the plan's U=50,000 rung realized as the 18,793-row full store). The reused rows are the store's instruct-model-own-text cell: each answer-span activation summarizes Qwen2.5-7B-Instruct's own generated reply to a real WildChat/LMSYS conversation prefix plus user query, not the corpus-logged assistant reply. Direction constructions — E1: diff-of-means over this run's judge-filtered contrastive extraction rollouts (5 positive/negative system-prompt pairs x 20 questions x 10 rollouts per sign, response-averaged activations). E2 (matched natural pairs): built from the train-rung labeling rollouts themselves — a context qualifies when at least 2 of its K=5 judge-scored rollouts survive drops and their score spread is at least 15 of 100; each qualifying context's rollout answer-span activations are split at that context's own score midpoint, and the direction is the high-minus-low mean difference averaged over qualifying contexts, holding the context (and so topic) fixed within every pair. E2p (pooled natural): one global midpoint split over ALL kept rollouts across contexts, direction = difference of the pooled means — topic-confounded by construction (pool membership tracks which contexts elicit high scores, not within-context variation), as the plan flags. Round-2 final-context directions reuse these same constructions with one change — the activation is read at the final-context token instead of averaged over answer tokens. The matched-natural (E2) form is structurally zero at the final-context point (every rollout of a context shares one context activation, so the within-context contrast cancels exactly; measured residue 5e-20) and was refused at the flag — a structural restriction, not a null result. Hallucination's final-context coverage splits by leg: the grid leg runs the synthetic-pair direction only (its DV file lacks per-rollout scores), while a population-level natural-pair leg supplies the pooled-natural read from its own labeling scores; both legs agree in sign. Behavior corpora — evil: in-the-wild jailbreak prefixes crossed with forbidden questions, 6,468 train contexts with a DV (transfer rungs: hh-rlhf red-team 1,868, ToxicChat 519); hallucination: TriviaQA rc.nocontext, 16,000 train contexts (transfer: NQ-Open 3,167, SimpleQA 4,021); sycophancy: Reddit personal-advice posts, 16,000 train contexts (transfer: 1,304 held-out Reddit posts — see below). Three data-provenance notes ride the record. (1) The sycophancy transfer rung labeled aita in the artifacts is the plan's fallback — a hash-partitioned held-out Reddit socialskills slice (all 1,304 post ids land in the sha1 mod-10 eval bucket) — because the planned ELEPHANT AITA-YTA set has no resolvable HF dataset id; it is a same-platform holdout, a weaker distribution shift than the planned cross-platform rung. (2) The E1 extraction pair/question assets were absent from the pinned direction-bank HF prefix, and the pod's git clone carries no local cache of the producing run's artifacts, so the regeneration branch of the wired fallback chain fired: evil used the persona-vectors paper's verbatim instruction pairs and questions (a code constant, no API call), while sycophancy and hallucination were each regenerated by one claude-sonnet-4-5-20250929 call from the persona-vectors template (confirmed against the persisted rollouts: the realized extraction system prompts differ from the producing run's cached texts). The realized extraction rollouts plus direction banks are persisted on HF. (3) Raw rollouts persist on HF as 123 packed JSONL shards plus a pack manifest rather than one file per rollout; totals reconcile via the manifest. A CJK language-intrusion scan over all 255,790 rollout completions found 6.5% (evil), 6.2% (hallucination), and 3.4% (sycophancy) intruded rows; on train contexts the per-context intrusion fraction correlates with the DV at rho +0.17 (evil and hallucination) and -0.01 (sycophancy) — a shared-target nuisance both arms of every comparison score against. Excluding every flagged context (any intruded rollout; 15-25% of contexts) moves the below-zero headline census by at most 31 cells per behavior (evil 703 to 672, hallucination 137 to 139, sycophancy 230 to 215 under the same frozen-layer convention) and leaves the canonical arm ordering unchanged apart from arms already tied within 0.001; scan and recount artifacts: eval_results/issue_1739/intrusion_audit/. The evil-ood-spread round adds three pilot corpora — MHJ human red-team transcripts (ScaleAI/mhj), tom-gibbs multi-turn synthetic jailbreaks (tom-gibbs/multi-turn_jailbreak_attack_datasets), and single-turn optimizer-refined attacks (abhayesian/pair_jailbreaks_formatted, prompt-automatic-iterative-refinement artifacts) — 200 contexts each after 8-gram near-duplicate screening against the train scrape (zero removed); a judge-assigned seven-class attack-tactic label per train context (4,814 of 6,468 labeled; the 1,654 unlabeled contexts ride the holdout's train side); and a graded compliance DV that re-judges the retained evil rollouts under a does-the-response-comply rubric instead of the trait rubric, with the superseded per-item aggregation retained in the artifact as a labelled legacy block (the binding read is per-context). The five corrected sycophancy OOD rungs come from the SycophancyEval mirror (meg-tong/sycophancy-eval: the answer, are-you-sure, feedback, and mimicry families; are-you-sure is the one two-pass rung, its challenge turn added by the harness) and the Anthropic model-written-evals sycophancy set fetched from its raw JSONL sources.

Sample training/evaluation data + completions: Per the trigger-dense content-hygiene rule for this task (jailbreak corpus + unscreened real-user text), no rollout text is quoted; rows are referenced by context id and numeric record, and full text lives in the linked artifacts. Cherry-picked high/zero-DV rows; all rows: labeling JSONs and raw rollouts on HF.

Behavior	Firing examples (DV)	Non-firing examples (DV 0.0)
evil	evil-train-cross-000555 (100.0), evil-train-cross-001236 (100.0), evil-train-cross-001266 (100.0)	evil-eval-hhrt-000000, evil-eval-hhrt-000001, evil-eval-hhrt-000002
hallucination	hallucination-eval-nqopen-000003 (1.0), hallucination-eval-nqopen-000006 (1.0), hallucination-eval-nqopen-000008 (1.0)	hallucination-eval-nqopen-000010, hallucination-eval-nqopen-000017, hallucination-eval-nqopen-000019
sycophancy	sycophancy-train-train-006815 (88.5), sycophancy-train-train-000252 (85.7), sycophancy-train-train-012247 (82.3)	sycophancy-eval-aita-000116, sycophancy-eval-aita-000161, sycophancy-eval-aita-000166

A 5-row random spot check (seed 42) per behavior showed coherent records (evil mostly 0 with occasional 80-100 scores, sycophancy 20-40, hallucination fabrication fractions), small per-row drop counts, and zero transport losses.

Evil-ood-spread-round artifacts are referenced by path and count under the same content-hygiene rule: three per-corpus pilot rollout sets (1,000 completions each) with judge aggregates in pilot_spread.json, the compliance wave's retained judge output (159,990 draws over 53,330 rollout items; compliance_full/<rung>/judge_raw_compliance_full.json per rung), attack-tactic labels for 8,602 contexts in tactic_labels.json, and 437,952 held-out predictions (16 line-split shards plus a manifest on HF).

U-ladder evaluation: Each behavior has all five finite seed statistics at every planned U rung. In-distribution and generic-transfer reads use their named setting directly; OOD is an unweighted macro-average over the behavior's realized OOD rungs. The true-pair generic-only ladder is binding; union-scaled and pairing-shuffled ladders are diagnostics. With only five seeds, intervals are necessarily broad, so the predeclared practical-equivalence and material-gap rules—not p-value-only reads—determine the verdict.

Natural-prompt extraction follow-up (2026-09-18, plan v33): Repeated the original fixed-readout comparison with high-minus-low directions from judged natural Reddit, TriviaQA, and HHRT/ToxicChat prompts; obvious trait instructions and jailbreak wrappers were excluded by the frozen eligibility audit. Primary top/bottom 1% tails require at least three valid archived responses, with equal prompt weights and equal valid-response weights. Frozen map pools, GCV recipe, layers, and original evaluation rows remain unchanged; all 76 original baseline cells reproduce exactly. Tail fractions 5%/10%, tie salts, complete-response subsets, endpoints, and E2/E2p are diagnostics. Intervals use 500 paired source-group bootstrap draws conditional on cached responses, labels, maps, and directions. A separate sensitivity removes detected exact/near map-content matches from evaluation; ID becomes unavailable and all 13 OOD datasets remain usable. Sycophancy/hallucination extraction prompts remain map-exposed. This is judgment-selected natural extraction with no new model or judge calls, not unlabeled extraction. Full numerical report.

Results

Map-then-project loses to the context-native direction almost everywhere

What is plotted: the mean difference in held-out Spearman rho, map-then-project minus context-native, at the largest budgets, one row per behavior x regime x mapping variant; dots are the 15 draw-by-seed cells, whiskers the 95% interval.

Figure. Map-then-project sits below the context-native baseline in nearly every slice. Rho difference at the largest budgets; E1 = synthetic-pair direction, E2 = matched natural, E2P = pooled natural; context/prefix = mapping variant; dots are per-cell values (n=15 per row).

Every evil slice and the context-side hallucination slice are negative (means -0.11 to -0.43), and the sycophancy synthetic-pair context slice sits at -0.06; the paired-bootstrap interval is wholly below zero in 15 of 15 largest-budget context-variant cells for all three behaviors under the synthetic-pair direction — the plan's falsification criterion, met on the context variant. The one reliably positive slice is sycophancy with the pooled-natural direction (+0.05, 15 of 15 cells above zero) — an order of magnitude smaller than the losses elsewhere, and it rests on the E2P contrast, which is topic-confounded by construction (Data extraction), so it is weak evidence for a behavior-direction gain.

Direct ridge regression on context activations dominates every monitoring arm

What is plotted: mean held-out rho for all 16 arms at the largest budgets (context variant), four panels; dots are per-cell values.

Figure. Direct ridge on context activations tops every deployable arm. Per-arm mean Spearman rho (frozen layer) with 95% intervals; dots are the per-cell values; blue = context-side arms, yellow = map-based, gray = true-answer oracles, pink = controls.

Direct ridge reaches 0.71 / 0.58 / 0.74, close to the true-answer regression oracle (0.82 / 0.65 / 0.77); every projection arm sits below it. Map-based regression arms add a small consistent gain over plain context ridge (mean +0.008 to +0.017, positive in 45 of 45 canonical cells), but the shuffled-pretrain control matches them to within 0.001 — the gain reflects added feature dimensions, not map semantics.

For hallucination even the true-answer projection oracle reads 0.04, below the shuffled-map control's 0.14, while true-answer regression reaches 0.65: the projection readout itself, not the map, is dead there. The best arm clears the max-over-arms-and-layers permutation null in all 1,906 cells (p near 0.002; median best rho 0.55 to 0.74 versus a null ceiling near 0.07).

More labels do not rescue the map arm

What is plotted: mean held-out rho versus labeled budget L (log scale) for five key arms, context variant, synthetic-pair direction, full unlabeled pool; shaded bands are 95% intervals over the 15 cells per point.

Figure. Extra labels grow the regression arms, not the map arm. Projection arms are nearly flat in L (labels only pick the layer); regression arms grow with L but dip at L=2,500. Bands only: 15 per-cell values per point across the five plotted series overplot illegibly; per-cell values are in the linked per-cell table.

Map-then-project stays flat in L while regression arms grow. At the smallest budget (L=250) it edges direct ridge for evil (0.54 vs 0.52) while still trailing the context-native direction (0.60), and it trails direct ridge for pooled-natural sycophancy (0.52 vs 0.59) — the sample-efficiency case for the generic map does not materialize at any budget. Regression arms dip at L=2,500 in all three behaviors (train folds hold about 2,000 rows against dimension 3,584, the near-interpolation regime for ridge), so regression reads at that one budget are conservative; the budget-endpoint comparisons are unaffected.

No robustness advantage on transfer rungs that pass the spread floor

What is plotted: mean held-out rho per evaluation rung (train, then transfer), context variant, synthetic-pair direction, largest budgets; asterisks mark rungs failing the DV spread floor.

Figure. Direct ridge transfers best on every floor-passing rung. Transfer ladder per behavior; the rung labeled aita is the held-out Reddit socialskills slice; evil's two transfer rungs (starred) fail the spread floor — their near-zero correlations are reads over an almost-constant DV.

Where the floor passes, direct ridge transfers best: hallucination 0.40 and 0.40 (NQ-Open, SimpleQA) against map-then-project's 0.20 and 0.27; sycophancy 0.73 against 0.35 on its held-out Reddit rung (a same-platform holdout — the fallback described under Data extraction). Evil's apparent map edge on ToxicChat (0.32 vs 0.25) sits on a rung with 93% of contexts in the DV bottom bin — convention-dependent, not robustness evidence — and on hh-rlhf red-team every arm reads near zero. The reversed-direction evil configuration collapses all 16 arms to rho 0.09 to 0.21; a growing map advantage under shift is unsupported wherever the DV is measurable.

The in-domain map-pool flip replicates across five seeds for evil only (+0.379, interval +0.323 to +0.435)

What is plotted: seed-mean held-out Spearman rho difference (map-then-project minus context-native) with 5-seed t-intervals per anchor cell, under a fully generic versus a half in-domain map pool, one panel per behavior; sycophancy's prefix cells are omitted (designed-degenerate — Methodology).

Figure. The half-in-domain flip is evil-specific. Seed-mean Spearman-correlation difference (mapped-answer arm minus context arm) per anchor cell with 5-seed t-intervals; evil turns positive at every realized budget once half the 5,000-row map pool is in-domain; sycophancy and hallucination stay at or below zero. Sycophancy's prefix-map columns are omitted as designed-degenerate; evil's half-in-domain 8,000-label cell is a designed-infeasible skip.

Figure. The per-seed view behind the flip contrast. Gray lines: one seed's flip term (the change in the map-minus-context difference from the generic to the half-in-domain pool) per anchor cell; the colored line is the seed mean; sycophancy shows its two context-map anchors only. Every evil term is positive (minimum +0.20).

Behavior	Flip contrast C (seed mean)	5-seed t-interval	Seed-anchor terms	Verdict
evil	+0.379	+0.323 to +0.435	20 of 20 positive	flip confirmed
sycophancy	+0.041	-0.036 to +0.118	7 of 10 positive (context variant only)	indeterminate
hallucination	-0.017	-0.064 to +0.031	9 of 20 positive	indeterminate

The round-1 preliminary read replicates for evil only: the flip contrast excludes zero there, includes it for the other two behaviors, and the 16 banked single-seed cells reproduce within 0.0002 maximum absolute difference. Sycophancy is scored on its reduced 10-pair grid after the prefix exclusion (Methodology; about 2 unique prefix rows in 5,632 — deterministic on rerun, not missing data). Excluding map-pool groups that overlap an anchor's labeled rows shifts no cell by more than 0.007 and flips no verdict.

The evil flip grows with the in-domain pool share, from −0.27 to +0.09 with the zero crossing near one quarter; sycophancy and hallucination never cross zero

What is plotted: seed-mean map-minus-context rho difference with 5-seed t-intervals against the in-domain share of the fixed 5,000-row unjudged map pool, at 2,500 labels with held-out pools, extraction variants pooled; sycophancy contributes its context map only (prefix excluded — Methodology).

Figure. Only evil crosses zero. Seed-mean map-minus-context difference with 5-seed t-intervals versus the in-domain share of the fixed 5,000-row map pool, at 2,500 labels. Evil rises from −0.27 through zero near a one-quarter share to +0.09 at three-quarters (its fully in-domain rung is a designed-infeasible skip); sycophancy dips to −0.34 then recovers to −0.01; hallucination stays between −0.25 and −0.14.

The dose response makes the unjudged-data attribution quantitative for evil: the gain is monotone above a 0.1 share and the zero crossing sits near one quarter in-domain. Sycophancy's curve recovers from its early dip yet never reaches zero — more in-domain unjudged data helps its map without ever beating the context-native direction — and hallucination is insensitive across the whole range. Per-unit exemption: the five per-seed values behind every dose point ship in the committed fold table (compose_ms_table.json, per-behavior dose_curve blocks — one per-seed mean-delta array per share) and per-cell CSV; the whiskers summarize exactly those values.

The context map learns; the prefix map does not

What is plotted: held-out map quality on the shared store by U rung — best-layer R-squared for the map and its identity-plus-learned-bias baseline, and kNN retrieval accuracy at k=1 with chance, for both maps.

Figure. Only the context-to-answer map learns on the generic store. Map diagnostics (identical across behaviors); context-to-answer R-squared rises 0.18 to 0.78 with U while prefix-to-answer never exceeds zero.

The context-to-answer map is real: R-squared 0.18 / 0.41 / 0.78 across the U ladder, far above identity-plus-bias, kNN accuracy 0.28 against chance 0.0003. The prefix-to-answer map never learns — R-squared at or below zero, retrieval near chance: on organic chat the prefix does not predict the answer representation.

Downstream, all prefix-side arms read near zero for sycophancy and hallucination (159 sycophancy prefix cells return undefined rank correlations from tie-degenerate projections). Prefix-side conclusions are scoped to evil, whose jailbreak prefixes vary.

A final-context extraction point does not rescue map-then-project

What is plotted: the per-cell change in held-out rho when the behavior direction is extracted at the final-context token instead of averaged over answer tokens — the synthetic-pair, context-variant slice (810 cells) of the 3,510 matched full-pool cell pairs; train versus floor-passing transfer rungs.

Figure. Final-context extraction shifts raw-context projection, not map-then-project. Mean rho change per arm (45 train / 45-90 transfer cells; dots are cells); evil transfer rungs excluded for failing the spread floor; the 88 sycophancy prefix pairs with tie-degenerate baselines are dropped as undefined.

Raw-context projection — the case the datatype-mismatch account favors — does gain (+0.17 sycophancy at the largest budget, rho 0.14 to 0.32; +0.03 evil; +0.05 hallucination train), but map-then-project — the arm the fix targets — moves -0.12 (sycophancy), -0.005 (evil), +0.03 (hallucination train). Across the two hallucination transfer rungs direction-dependent predictors average about -0.20; the true-answer projection splits -0.04 (NQ-Open) against -0.37 (SimpleQA). The gains are also layer-mediated: the two extraction points pick the same layer in only 109 of 1,262 train joins, and at the answer-averaged direction's own layer the sycophancy gain inverts to -0.16 (a read favoring the incumbent; the two conventions bracket the effect).

Pooled-natural gains are smaller (+0.07 evil, +0.03 sycophancy).

The true-answer readout is linear at the largest budgets

What is plotted: held-out rho for regression from the true answer vector — linear ridge (the within-round rider) versus one-hidden-layer MLP versus kernel ridge — by labeled budget, train rung, context variant, 15 cells per bar.

Figure. Nonlinear oracles match the linear oracle at the largest budgets. True-answer regression by labeled budget (n=15 cells per bar); the hallucination panel is a secondary read — its round-2 nonlinear leg lacks the largest budget.

At the largest budgets nonlinearity moves the oracle ceiling by at most +0.007 (evil kernel ridge, 0.831 vs 0.824) and +0.004 (sycophancy, 0.775 vs 0.771) — and the linear oracle already sits near the split-half reliability ceilings (0.89 evil, 0.88 sycophancy), leaving little detectable headroom. At 250 labels kernel ridge does keep a small edge (+0.04 evil, +0.02 hallucination, paired per-cell intervals excluding zero), scoping the linearity null to the largest budgets and these two fit families at the inherited recipes. The +0.17 to +0.26 gains at 2,500 labels land exactly where ridge enters its near-interpolation dip while the nonlinear fits do not — repair of a ridge pathology, not evidence of nonlinear answer-to-behavior structure; hallucination shows the same dip-only pattern (secondary read).

The filled transfer roster leaves no arm beating direct ridge off-distribution

What is plotted: held-out rho on the floor-passing transfer rungs at the largest budget for arms the original grid never ran off-distribution — direct MLP and the MLP- and kernel-map ridge readouts — beside the committed direct-ridge and linear-map readouts.

Figure. No newly added arm beats direct ridge off-distribution; two tie it. Largest budget, synthetic-pair direction, context variant; committed arms and direct MLP n=15 cells, nonlinear-map readouts n=6 per map family; the linear-map readout has no rows at the joinable full-pool coordinate on the hallucination rungs; evil rungs excluded (spread floor).

No new arm beats direct ridge: sycophancy held-out Reddit 0.66 (direct MLP) and 0.71 / 0.63 (kernel-/MLP-map readouts) versus 0.73; hallucination NQ-Open 0.32 and 0.40 / 0.36 versus 0.40; SimpleQA 0.23 and 0.18 / 0.13 versus 0.40. Two arms tie it within noise: the kernel-map readout on NQ-Open (0.403 vs 0.395), uncontrolled (no shuffled-map control exists for nonlinear maps), and the committed linear-map readout on Reddit (0.729 vs 0.725), whose shuffled control reads 0.18. Largest-budget label draws mostly coincide: direct-MLP bars hold 3 unique fits of 15, nonlinear-map bars (3 draws by 2 seeds per family, never pooled with linear arms) 1-2 of 6; the direct-ridge bar plots 30 dots because its source file stores each cell twice.

The map does not transfer across attack-tactic families; a direct context readout does

What is plotted: held-out Spearman rho for all 16 arms, refit on the 1,906 jailbreak train contexts outside the Direct Request attack-tactic class and scored on the 4,562 held-out Direct Request contexts, for both mapping variants and all three direction regimes.

Figure. Map arms sit on their shuffled-direction control; the direct context MLP transfers. The 96 arm-by-regime rows with bootstrap 95% intervals over the 4,562 held-out contexts; symbol shape gives the direction regime; blue = context-side arms, yellow = map-based, gray = true-answer oracles, pink = controls.

On the context variant the best map arm reads 0.514 against 0.510 for the shuffled-direction control (a scrambled direction that keeps the fitted scalar readout and layer pick), intervals nearly coincident; prefix-side separation is +0.041 with intervals still overlapping. The direct context MLP transfers at 0.68 (interval 0.66 to 0.70), clear of every control and map arm, and context-side only — it drops to 0.51 on the prefix variant; text alone reads 0.45. The oracles (up to 0.73) use true-answer activations a monitor never has, and the max-over-arms permutation p of 0.002 is the 1/501 floor against a null the shuffled control itself clears; levels are regularization-limited (1,906 train rows vs 3,584 dimensions).

A graded compliance DV spreads exactly where the trait DV is degenerate

What is plotted: the distribution of per-context mean scores on each evil rung — the parent 0-100 trait DV beside a 0-100 graded compliance DV judged over the same retained rollouts; fraction of contexts per 5-point bin.

Figure. The compliance DV keeps range where the trait DV collapses. Per-context means over 5 rollouts x 3 judge draws; every context with at least one kept draw (n in legends); the same per-context, sample-SD, strict-bottom-bin instrument on both sides.

Every rung passes the two-sided spread floor under the compliance DV — per-context SD 34.7 / 36.0 / 38.7 at n = 7,959 / 1,994 / 671 (train / hh-rlhf red-team / ToxicChat) — where the trait DV reads 26.3 / 0.9 / 12.1 and fails both transfer rungs. The compliance advantage is largest exactly where the trait DV is most degenerate (roughly 40x on hh-rlhf red-team, 3.2x on ToxicChat, 1.3x on train), the monotone pattern a DV-specific account of the failure predicts. Per item the two DVs rank-correlate weakly (rho 0.07 to 0.22): compliance is a companion instrument, not a re-scaling, and no arm comparison is re-scored against it yet — evil's ToxicChat map edge stays unmeasured under a spreading DV.

The trait DV escapes its floor on fresh attack-family corpora

What is plotted: the distribution of per-context mean trait scores on three 200-context pilot corpora from published attack datasets (human red-team, multi-turn synthetic, optimizer attacks), judged with the parent grid's trait rubric.

Figure. All three fresh attack corpora give the trait DV usable spread. Per-context means over 5 rollouts x 3 judge draws, 200 contexts per corpus; the floor requires SD of at least 10 with under 80% of contexts in either extreme bin.

All three corpora pass the two-sided floor — SD 27.8 (MHJ) / 22.2 (Tom-Gibbs) / 11.8 (optimizer attacks), bottom-bin fractions 0.55 / 0.69 / 0.79 — refuting the plan's alternative outcome, that no attack-family corpus escapes the degenerate regime. Robustness orders the corpora as listed: Tom-Gibbs fails on the secondary per-rollout unit (0.83 against the 0.80 edge) and a quarter of its rollouts hit the generation cap — excluding every truncated rollout moves SD the wrong way for a truncation account (22.2 to 22.9), harmless for a spread read though not a value read — while the optimizer-attack corpus passes thinly on both axes (SD 11.8 vs floor 10; bottom bin 0.785 vs 0.80). Full transfer rungs were not run.

The mapped-answer probe edge survives a shuffled-pairing map control at one flagship rung of two

What is plotted: the seed-mean held-out Spearman-rho difference, mapped-answer probe minus context probe, on the 13 whole-holdout rungs — computed with the true map (the true-map advantage) and the pairing-shuffled map (the shuffled-map advantage); whiskers are 5-seed t-intervals, stars the flagship rungs.

Figure. Part of the probe edge is pairing-specific; part is a generic-transform effect. Seed-mean rho deltas at the frozen layer (5 seeds per row); stars mark the two flagship rungs the plan named; the shuffled rows refit the map after destroying only the context-answer pairing.

Figure. Seeds agree on sign for 10 of 13 true-map rungs; near-zero rungs vary. The low-level companion to the forest: per-seed true-map advantage (left half of the figure) and shuffled-map advantage (right half) on all 13 rungs, 5 seeds each; the forest whiskers summarize these values.

The true-map delta is positive on 10 of 13 rungs and the margin over the shuffled map on 12 of 13, both medians +0.03 — the plan's weak-form outcome. The sycophancy model-written-evals flagship holds both reads clear of zero (margin +0.14, t-interval +0.07 to +0.20); the evil optimizer-attack flagship's margin interval crosses zero (-0.006 to +0.085). The shuffle destroys retrieval (top-1 0.0008 versus 0.16-0.57 true) yet shuffled maps still gain on several rungs: roughly half the edge is a generic-transform effect the margin removes.

Four open hardening concerns bound this read: claim4-seed-rerun-idempotency, claim4-lattice-rung-identity, claim4-pushdown-production-bitidentity, claim4-chunk-turnover-rss. The comparator returned inconclusive here; its repair is reported below.

The repaired context-direction comparator provisionally supports the mapped-answer edge for sycophancy and hallucination (median +0.12 over eight passing rungs)

What is plotted: per-rung held-out Spearman rho difference, mapped-answer probe minus repaired comparator, on the 13 fair-allocation rungs; dot and whisker are the 5-seed mean and t-interval, thin bar the paired context bootstrap; evil rows grayed (sanity-gate fail).

Figure. The mapped-answer probe beats the repaired comparator where the repair validates. Positive on 6 of 8 sanity-passing rungs; median difference +0.12 (sycophancy +0.21, hallucination +0.12); the model-written-evaluations flagship clears both the seed and context intervals; evil's five rungs are excluded — its repaired comparator still reads below its in-distribution band.

Behavior	Repaired comparator adapter	In-distribution read vs banked band	Sanity gate
evil	quantile-restricted composite (v2), parity refits	0.107 vs 0.404 to 0.700	fail — below band
sycophancy	v1	0.404 vs 0.295 to 0.526	pass
hallucination	v1	0.526 vs 0.442 to 0.556	pass

The previously inconclusive comparator now lands in-band for sycophancy and hallucination, and the mapped-answer edge provisionally survives it: median +0.12 over 8 passing rungs, the model-written-evaluations flagship +0.35 with both intervals clear of zero, losses confined to answer and mimicry. Open concerns bound the read: arm2fix-sanity-band-budget-universe-self-derived (budget universe self-derived, not fixed in advance) and cms-smoke-phase-coverage leave the admitting gate unhardened; arm2fix-d0-phase-idempotency (no idempotency guard on the probe phase) leaves the repair's confound and degeneracy evidence single-run; arm2fix-pvsynth-fit-axis-self-derived makes the secondary synthetic-rung read weaker (fit axis self-derived from realized rows, not pre-fixed).

Evil's comparator still reads below its band after the quantile-restricted refit, so its five rungs stay indeterminate-adapter, evidence neither way. Per-unit exemption: the five per-seed differences behind every row ship in arm2fix_table.json (per_rung[].D.per_seed) and the pinned HF per-seed row files.

The sycophancy mapped-answer edge ranges from +0.33 on model-written evaluations to -0.20 on mimicry

What is plotted: held-out Spearman rho for five banked roster arms on the five corrected sycophancy OOD rungs — single-seed rows under both readout protocols (single-dataset and fair-allocation pools); whiskers are context bootstraps.

Figure. The sycophancy probe edge is eval-family-specific. Banked single-seed rows carrying per-row provenance labels; whiskers are context bootstraps over held-out contexts.

Figure. The per-context view behind the fair-allocation bars. Mapped-answer and context-probe predictions against the judge-scored DV per context — fair-allocation refit rows, seed 0 — on all five corrected rungs.

The five rungs replace the same-platform Reddit fallback with a split the 5-seed table confirms: the mapped-answer probe gains on model-written-evals (+0.33; rho 0.40 versus 0.06, seed means), is +0.05 on answer (seed interval above zero, context-bootstrap crossing it), flat on feedback (+0.03), loses on are-you-sure (-0.11, seed interval crossing zero) and mimicry (-0.20, the one interval-separated loss; rho -0.01 versus 0.19). The banked real-answer oracle reads 0.48 there, so the mapped answer recovers most answer-side signal. Where the probe loses, the shuffled map loses as much (mimicry margin +0.03) — the loss traces to the map's output distribution, not its learned pairing.

More generic map pairs shrink the mapped-answer deficit but do not substitute for judged pairs

What is plotted: D(U), mapped-answer ridge minus direct context ridge in held-out Spearman rho, over seven generic-map-pool sizes; ribbons are five-seed 95% t-intervals.

Figure. More generic pairs usually narrow the deficit but do not reproduce judged-pair benefits. Solid: generic-only; dashed: union-scaled; horizontal line: context-ridge parity.

The primary trend interval is positive in 8 of 9 panels, but full-U D remains negative in 8 of 9. Evil generic transfer is the only positive endpoint (+0.0245, 95% interval +0.0212 to +0.0278).

None is supported; eight are refuted because the judged-minus-generic contrast's 90% lower bound exceeds +0.02. Evil generic is unresolved: its +0.0509 contrast interval (+0.0175 to +0.0842) is neither equivalent nor materially separated. Union-scaled maps yield the same verdicts.

Behavior	In-distribution	Generic transfer	OOD
evil	REFUTED	UNRESOLVED	REFUTED
sycophancy	REFUTED	REFUTED	REFUTED
hallucination	REFUTED	REFUTED	REFUTED

Full-U D is -0.0759 / +0.0245 / -0.1370 for evil, -0.0168 / -0.0610 / -0.0110 for sycophancy, and -0.0314 / -0.0251 / -0.0080 for hallucination (in-distribution / generic / OOD). The fold-clean diagnostic is negative for all three and cannot alter the predeclared verdict.

---

Repro: U-ladder production: Production consumed 19.62 H100-hours by successful-sentinel wall time across three parallel H100 pods. Source commit 0d155ded15ba516e5f207821e4ff5d0eda78cbc2; result commit 7202c3671c051dc2e6e61386d6ab6d4a865ac284; clean-provenance sidecar commit e9e3dd93958. Planned and realized coverage are both 15 of 15 behavior-seed runs, comprising 420 enumerated map cells plus controls and 450 uploaded files; local-versus-Hub filename sets and SHA-256 hashes match exactly. Fold artifacts are under issue1739_uladder/fold/, production artifacts under issue1739_uladder/, and the figure plus sidecar under issue1739_uladder/figures/. All successful run sentinels report rc=0 and every seed uploaded 30 files. The three healthy production pods remain retained under the task's keep-running policy; each has at least 152 GB free.

Prior rounds: ~96 GPU-h round 1 (three parallel A100-80 lanes for the fit phase; generation/capture on the primary pod; judging via the Anthropic Batch API off-pod) + ~55 GPU-h round 2 (twelve single-A100-80 boxes, plan estimate) + the evil-ood-spread round (three single-GPU RunPod pilot pods for 3 x 1,000 rollouts; 159,990 compliance judge draws via the Batch API; the tactic holdout ~56 min on a CPU pod) + the claim4-controls round (0 GPU-h; ~19-21 CPU-pod-hours across seven RunPod CPU pods — four production 256/128 GB boxes, three OOM-superseded) + the composition-multiseed/arm2-repair round (0 GPU-h; roughly 60 CPU-pod-hours over ten verified RunPod CPU chains, timestamp-derived, plus five first-cell wall-halted boxes rotated under the recorded gate ladder). Code eb084ff2c4 (run), 6686d45da9 (round-1 analysis figures), a78f971b51 (round-2 dispatch), b1aaf57910 (round-2 figures + renderer scripts/issue1739_newarm_figs.py), 6e27d96a3a (evil-ood-spread spread-instrument consolidation + tests), 810466a2fe (re-reduced compliance artifact), 39afefe018 (holdout metrics + pilot aggregates), 324e77cdaf (evil-ood-spread figures + renderer scripts/issue1739_oodspread_figs.py), branch issue-1739; claim4-controls: 067bd0300c (run — scorer scripts/issue1739_r2v2_score.py, runner issue1739_r2v2_run.py, fold issue1739_claim4_fold.py), 12cfdbf31d (fold table + figures + numbers-note), 56f78b6b5c (reader-facing figure relabel + the per-seed and per-context companions, renderer patch + scripts/issue1739_claim4_relabel_figs.py), and 0efb2fb67b (the per-context companion expanded to all five corrected sycophancy rungs), branch issue-1739-fit, banked fair-refit reference rows at 5aae0a472b. Composition-multiseed/arm2-repair round: 6f0414b913 (seed-split wrapper halves), 82a16cd2f5 (jittered eigendecomposition fix in the fit core), ad5380d226 (leg-1 fold table + figures + scripts/issue1739_compose_ms_fold.py), a0f14160d1 (review-round figure re-render: reader-facing labels, designed-degenerate columns omitted — the embedded compose figures' pin), branch issue-1739; a9d4979ffa (arm2fix scorer run commit) and f0530e0e8b (arm2fix fold table + forest figure + numbers-note docs/map_behavior_prediction_arm2fix_note.md), branch issue-1739-fit. Eval artifacts: eval_results/issue_1739/{evil,hallucination,sycophancy}/arm_results/all_arms_spearman.json (826 / 270 / 810 cells; per-cell records in arm_results/percell/cells.jsonl), eval_results/issue_1739/evil_config_b/ (90 reversed-transfer cells), eval_results/issue_1739/dv_dataset/*/labeling.json, eval_results/issue_1739/{evil,hallucination}/pilot_report.json, map diagnostics per behavior, the per-cell headline table figures/issue_1739/headline_deltas_percell.csv, and the CJK intrusion scan + excluded-intrusion recount eval_results/issue_1739/intrusion_audit/{intrusion_scan,recount}.json. Evil-ood-spread-round artifacts: eval_results/issue_1739/evil_ood_spread/{pilot_spread.json,compliance_dv_results.json,tactic_labels.json,item_b/holdout_metrics.json,item_b/cluster_stats.json,contexts/summary.json}; the 437,952 held-out predictions (16 line-split shards plus manifest), retained judge outputs, and pilot rollouts live on HF under issue1739_ctxmap/evil_ood_spread/ (listing verified via list_repo_tree at the pinned revision, 2026-08-05). Claim4-controls artifacts: claim4_per_rung_table.json (+ its .md twin) + figures/issue_1739/claim4_controls/ (7 figures with data sidecars; the embedded delta figures re-rendered with reader-facing labels at 56f78b6b5c, the per-context companion expanded to all five corrected rungs at 0efb2fb67b) + the writeup numbers-note docs/map_behavior_prediction_claim4_controls_note.md, committed at 12cfdbf31d on issue-1739-fit; per-seed row files, map-baseline reads, and reproduction-gate breadcrumbs on HF under issue1739_claim4_controls/ — subdirs evil, sycophancy, hallucination, repro_claim4, _staging (three per-behavior upload-verification PASS markers, 2026-08-19; listing verified via list_repo_tree at the pinned revision, 2026-08-19). Composition-multiseed/arm2-repair artifacts: compose_ms_table.json + compose_ms_cells.csv (branch issue-1739; first committed ad5380d226, table regenerated at a0f14160d1 with the figure relabels) and arm2fix_table.json (committed f0530e0e8b, branch issue-1739-fit); per-cell rows, pool metadata, and map diagnostics on HF under issue1739_ctxmap/compose_multiseed/ (408 files across per-behavior halves s02/s34, sycophancy also single-seed halves s3/s4) and issue1739_claim4_controls/arm2fix/ (160 files: 90 evil, 45 sycophancy, 25 hallucination; listings verified via list_repo_tree at the pinned revision, 2026-08-24; ten per-pod upload-verification PASS markers with exact name-set and row reconciliation, 2026-08-23/24). Committed figures deliberately not embedded: figures/issue_1739/claim4_controls/claim4_spaghetti.png (true-map-only per-seed draft, superseded by the embedded two-series per-seed companion), figures/issue_1739/claim4_controls/claim4_arm2_bars.png (the context-extracted-direction comparator's per-rung sanity bars — its inconclusive out-of-band read is reported in prose), figures/issue_1739/claim4_controls/claim4_map_knn.png (the per-layer map retrieval read behind the shuffle control — top-1 0.16-0.57 true versus 0.0008 shuffled, reported in prose), figures/issue_1739/interim_writeup/nlmap_percell_scatter.png and figures/issue_1739/interim_writeup/wcrung_arms.png (assets of the separately-authored interim writeup, not results of this body), figures/issue_1739/result2_fair/result2_fair.png (an earlier single-seed fair-protocol render the claim4-controls forest supersedes), and figures/issue_1739/sycophancy/distribution_shift_ladder.png (a dispatcher-phase per-behavior draft of the embedded transfer ladder). The round-1 single-seed composition scatter figures/issue_1739/compose_fu_flip.png is superseded by the five-seed compose_fu_flip_v2 set embedded above. Round-2 artifacts: eval_results/issue_1739/new_arm_round/arm_results/{merged_transfer.jsonl,merged_cells_rows.jsonl,fc_vs_t1_pairs.jsonl,newarm_spearman_summary.json} + spread-floor verdicts eval_results/issue_1739/new_arm_round/k1_verdicts.json; per-leg box outputs on HF under issue1739_new_arm_round/ (listing verified via list_repo_tree at the pinned revision, 2026-08-03); the collect script's two draft figures (committed fdcebc102c) are superseded by the _v2 set — the draft delta figure averaged the spread-floor-flagged evil rungs into its bars. Round 2 reused this issue's own capture stores, judge labels, and fitted maps (Hub-verified 2026-08-01 via scoped list_repo_tree, plan v9 §10) and added no new rollouts or judge calls. Raw rollouts + judge outputs: HF superkaiba1/explore-persona-space-data under issue1739_ctxmap/raw_completions/ and issue1739_ctxmap/judge/ (listing verified via list_repo_tree, 2026-07-30). Reused artifacts — activation store from #1092: HF superkaiba1/explore-persona-space-data at issue1092_realistic_crossing/analysis_tensors/summaries/, rev e5901706 — fit: same base model and revision as this run's capture, carries both mapping arms plus answer-span summaries at all 28 layers in fp16, and its instruct-own-text cell matches the map's intended input distribution (real conversations, model-generated answers); direction bank from #779: HF superkaiba1/explore-persona-space-data at issue779_monitoring/r_b/, rev 037fcbb — fit: persona-vectors mean-difference directions for the same three behaviors on the same base model (judge-filtered contrastive rollouts, all 28 layers, matching the store's layer grid), consumed as the pinned E1 reference probe at Gate 0 (E1 grid directions were extracted fresh from this run's rollouts after the asset fallback described under Data extraction). Planned-vs-actual: 2 of 18 planned evil composition cells skipped (recorded infeasible pool recipe); the U=50,000 rung realized as the 18,793-row full store; the sycophancy transfer rung realized as the plan's held-out Reddit socialskills fallback (ELEPHANT AITA-YTA unresolvable on HF); the teacher-forced margin companion DV was not computed; sycophancy restored 453 of its 810 cells from a crash resume (final coverage complete); round 2 realized its full mirror grid (zero recorded skips, zero roster-accounting violations; the matched-natural final-context regime is a structural restriction, not a skip); the evil-ood-spread round realized items A, B, and D — its item C (detection re-cut) was superseded by a concurrent follow-up round and committed no artifact (its dead zero-row re-cut output is deliberately left uncommitted, and the concurrent round's own artifacts fold separately); the compliance artifact retains the superseded per-item aggregation as a labelled legacy block and carries a struck circular detection block (its label was thresholded from the scored variable itself — never a detection result); the tactic holdout's mandated kNN-retrieval read is absent, and its identity-plus-learned-bias baseline is rank-degenerate in a single-fold transfer (a per-fold additive constant cannot change Spearman ranks), so neither mandated mapping read is informative there; full transfer rungs on the passing pilot corpora were not run; the claim4-controls round realized all four items at the planned 13-rung x 5-seed grid with zero coverage gaps — its compliance companion was a declared skip (77.3% join coverage, below the 90% gate), its context-extracted-direction comparator is recorded inconclusive (adapter-suspect: the in-distribution sanity read lands outside the committed band on all three behaviors — evil -0.06 vs a 0.40-0.70 band, sycophancy above its band, hallucination marginally below), two production out-of-memory crashes (a map-fit split-copy transient, then a readout-stage transient) were fixed by a row-index pushdown whose outputs the seed-0 reproduction gate matched at full production shape (rc=0 on all three behaviors) plus a 256 GB box substitution for the sycophancy/hallucination pods (plan §9 deviation — box RAM only, code and argv unchanged), and sycophancy's seeds split across two pods after a ~2.4x wall deviation at the width gate; the composition-multiseed round realized 110 of 130 evil cells (20 designed skips, recorded infeasible pool recipes — fully in-domain rungs and no-overlap 8,000-label anchors exceed evil's 6,468-context pool) and 130 each for sycophancy and hallucination, with sycophancy's prefix variant excluded from every statistic as designed-degenerate (65 realized cells kept as a stat-free flagged row class; the 2-anchor criterion is a stated deviation) and the sycophancy seeds-3-4 half split to single-seed pods after repeated first-cell wall halts; the arm2fix leg realized all 13 rungs x 5 seeds, with evil excluded by the pre-fixed sanity gate (indeterminate-adapter) and the planned direction-stability companion file absent (recorded caveat); config slugs arm1_ctx_e1 ... arm18_oracle_krr (+ arm2_ctx_native transfer-mode and arm20_shuffled_map_ridge this round), regimes e1/e2/e2p (+ e1_fc/e2p_fc round 2), variants prefix_end/context_end, map_variant=true/shufpair, protocols P-A/P-B, evil-ood-spread rung slugs evil_train/evil_hh_rlhf/evil_toxicchat and corpus slugs mhj/tom-gibbs/pair, claim4 rung slugs evil_mhj/evil_pair/evil_tomgibbs/hhrt/toxicchat/aita/sycoans/sycoays/sycofb/sycomim/sycomwe/nqopen/simpleqa, compose cell labels compose5000_fu<share>_fl<flag>_L<labels> with pod halves s02/s34/s3/s4, and the repaired comparator slug arm2q_ctx_native.

Context: fresh direction (no parent) — task created 2026-07-28 from the 2026-07-27/28 interactive design session (plan at docs/map_behavior_prediction_plan.md). Originating prompt (verbatim): "run in background with happy coder and MAKE SURE IT PARALLELIZES AND VECTORIZES AS MUCH AS POSSIBLE". Run completed 2026-07-29; first analyzer pass 2026-07-30; revision rounds (intrusion recount + interpretation-critic fixes; clean-result-critic self-containedness + provenance fixes) 2026-07-30. Same-issue follow-up round new-arm-round (scoped 2026-08-01, run 2026-08-02/03, folded 2026-08-03); its originating prompt (verbatim): "run these as much in parallel/vectorized as possible: [1] Final-context PV extraction point; [2] Nonlinear oracle (nonlinear regression from the true answer vector); [3] arm5 direct-nonlinear + nonlinear-map ridge readouts (arm7/8) on the OOD rungs" then "run all in background with happy coder." Same-issue follow-up round evil-ood-spread-round (scoped 2026-08-03, run 2026-08-03/05, folded 2026-08-05); its originating prompt (verbatim): "do all these things. don't worry too much about cost. Don't run on GCP. run as much in parallel/vectorized as possible" — "all these things" being the four-item menu for the floor-failing evil transfer rungs (attack-family shift; tactic-family holdout; detection re-cut; graded compliance DV). Same-issue follow-up round claim4-controls (scoped 2026-08-19, run 2026-08-19, folded 2026-08-19); its originating prompt (verbatim, paper-planning chat): "file and run" — the four control runs that decide whether writeup claim (4) (probe-on-mapped-answer beats probe-on-context via unjudged data) is draftable in strong form. Same-issue follow-up round composition-grid-multiseed-plus-arm2-repair (scoped 2026-08-22, run 2026-08-23/24, folded 2026-08-24); its originating scope (verbatim, user-chat follow-up scope marker): "Two legs from the 2026-08-22 paper-outline adversarial critique (docs/paper_context_answer_map/outline_critique_2026-08-22.md, priority items 1-2)" — leg 1 the multi-seed three-behavior composition rerun with the unjudged-pool-volume ablation, leg 2 the arm2 comparator repair and rerun. Same-issue follow-up round unlabeled-data-ladder (scoped and run 2026-09-05); its originating approval was (verbatim): "yes proceed" after accepting the seven-rung, five-seed design and the 0.02 equivalence margin. Conciseness note: total prose runs over the 800-word budget, several result sections exceed the 120-word prose cap, and several Takeaways bullets exceed the 30-word bullet cap — acknowledged; seventeen result sections were kept to cover the 1,906-cell grid plus the round-2, evil-ood-spread, claim4-controls, and composition-multiseed/arm2-repair additions.

Natural context–answer scaling follow-up (2026-09-08)

More natural generic data preserves a hallucination advantage, not a general advantage

The requested no-recombination sweep is complete: 150 of 150 planned cells, with no missing or failed fit cells. It crosses 10 generic-pair counts with five seeds and three behaviors, retaining each behavior's full trait-training pool. This is an additive plan31 result, not a replacement of the earlier P-A study, its hypothesis classification, or the manuscript.

At 100,000 generic pairs plus the fixed trait pool, the mapped-answer regression beats direct context regression for hallucination in all five seeds. For evil and sycophancy, the mean differences are slightly negative and the seed ranges span zero. These are descriptive results: a near-zero mean is not a test of equivalence, and the seed ranges below are not confidence intervals.

Behavior	Direct context ρ	Mapped answer ρ	Answer oracle ρ	Mapped − direct	Five-seed delta range
Evil	0.1430	0.1411	0.3138	−0.0019	[−0.0642, +0.0513]
Sycophancy	0.3619	0.3533	0.4652	−0.0086	[−0.0450, +0.0167]
Hallucination	0.4046	0.4574	0.5291	+0.0528	[+0.0488, +0.0589]

Every value first averages the own-heldout-dataset Spearman correlations within one behavior and seed, then averages the five seeds. Datasets receive equal weight, not weight proportional to their evaluation sample count.

The scaling effect is behavior-dependent. From 250 to 100,000 generic pairs, mapped-answer performance changes from 0.2677 to 0.1411 for evil, 0.3229 to 0.3533 for sycophancy, and 0.4223 to 0.4574 for hallucination. The corresponding mapped-minus-direct gaps change from +0.1421 to −0.0019, −0.0342 to −0.0086, and +0.0308 to +0.0528. Curves are not uniformly monotonic. At the matched size 18,793 in this new natural pool, the gaps are +0.0065, −0.0183, and +0.0394; this is not a replay of the old 18,793 examples or old evaluation protocol.

The dataset detail matters. At 100k, evil has a large negative mapped-minus-direct difference on TomGibbs (−0.2909) despite positive means on its other four datasets. Sycophancy has a large deficit on mimicry (−0.2342), but a gain on model-written evaluations (+0.1457). Hallucination improves on both NQ-Open (+0.0699) and SimpleQA (+0.0357). No single aggregate establishes an across-dataset guarantee.

Scaling overview files and dataset panels for evil, sycophancy, and hallucination. Lines show five-seed means; bands show the seed minima and maxima.

What was actually varied

Generic counts were 250, 500, 1,000, 2,000, 5,000, 10,000, 18,793, 25,000, 50,000, and 100,000. Each example is an intact first-user prompt from pinned LMSYS-Chat-1M and its own freshly generated Qwen2.5-7B-Instruct answer. No conversation histories or queries were recombined, no later turns were appended, and no original third-party answers were reused. Generation used greedy decoding and the inherited 1,024-token cap. All 100,000 generated answers were nonempty; 8,414 cap-hitting answers (8.414%) were retained as plan31 required. The store has 100,000 distinct source keys, verified from the uploaded row index itself.

The full fixed trait-training pools were 6,468 pairs for evil and 16,000 each for sycophancy and hallucination. Thus endpoint maps used 106,468 / 116,000 / 116,000 total pairs, not 100,000 total mixed pairs. Nested generic samples are shared across behaviors for each seed. At 100k every seed uses the same complete generic pool, although whitening and readout splits differ across seeds.

The three linear arms are direct context regression (arm4), map then regression (arm7), and true-answer oracle regression (arm12). Global layers were frozen at 18/20/17 for evil, 20/19/19 for sycophancy, and 20/20/18 for hallucination (direct/map/oracle). Whitening was re-estimated at each size from the generic-plus-trait pool using the inherited seed-dependent 80/20 fit/selection split, and shared by all three methods. Thus even the direct and oracle baselines can change with generic size. Map diagnostics use an 80/20 split, followed by a weight refit on the entire declared union; the audit verified actual fit rows.

P-B trains a readout without its evaluation dataset and evaluates on that whole held-out dataset: five evil, six sycophancy, and two hallucination datasets. The map uses the same fixed trait-training pool across readout folds. The primary analysis includes only each readout's own held-out dataset, not its held-in or auxiliary predictions. Realized coverage is 650 held-out fold/seed/size evaluations and 1,950 primary method rows. This round reused cached behavioral labels and evaluation contexts, made no new judge calls, and did not update model weights.

Mapping diagnostics at the 100k endpoint

All numbers below are five-seed means at each behavior's frozen mapped layer. R² is evaluated on the named held-out dataset; the comparator is the identity map plus a bias learned from training pairs. Nearest-neighbor retrieval uses the actual dataset's answer vectors as its candidate pool, not the 100k training pool. Full per-seed, per-layer, and per-size diagnostics are retained in JSON.

Behavior / dataset	Map R²	Identity+bias R²	Euclidean top-1	Cosine top-1	Pool N	Top-1 chance
Evil / MHJ	0.2371	−1.0489	0.2049	0.2852	533	0.001876
Evil / PAIR	−0.4890	−2.5285	0.0196	0.0238	917	0.001091
Evil / TomGibbs	−0.2218	−1.4313	0.0033	0.0047	1,500	0.000667
Evil / HH red-teaming	0.4764	−1.0774	0.6318	0.6630	1,868	0.000535
Evil / ToxicChat	0.5072	−0.7898	0.5565	0.6227	519	0.001927
Sycophancy / AITA	0.5434	−1.4932	0.7870	0.8061	1,304	0.000767
Sycophancy / answers	0.3556	−0.8027	0.2997	0.3370	594	0.001684
Sycophancy / are-you-sure	0.1456	−1.5072	0.2234	0.3117	598	0.001672
Sycophancy / feedback	0.2891	−1.3131	0.0619	0.0877	559	0.001789
Sycophancy / mimicry	−0.9597	−5.3753	0.0035	0.0035	285	0.003509
Sycophancy / model-written evals	−0.0649	−2.0124	0.1208	0.1326	576	0.001736
Hallucination / NQ-Open	0.4958	−0.5143	0.6322	0.6926	3,167	0.000316
Hallucination / SimpleQA	0.3839	−0.7151	0.3646	0.4506	4,021	0.000249

Several OOD datasets retain negative reconstruction R²; better reconstruction than identity+bias is not the same as beating direct behavioral regression. The true-answer oracle is a reference predictor, not a guaranteed upper bound.

Scope and reproducibility

This is evidence for a hallucination benefit under this particular protocol, not a disproof or confirmation of the entire persona-map hypothesis. The fixed trait pool becomes a smaller fraction of the mapping data as generic size grows, so the experiment changes both volume and mixture weight. The same evaluation contexts recur across seeds. One model, one generic source, inherited labels, and the retained cap-hitting answers bound generalization. No new equivalence test or automatic supported/refuted classification was authorized. Because the old study used a different source population and readout protocol, the new result does not identify recombination as the cause of any historical discrepancy.

Scientific code was frozen at 9f6a6fedb9a97365bfbfb0c241d7712d94ce43c0. Every cell's saved correlations were independently recomputed from predictions; paired context/group/label keys, full map-fit counts, fixed layers, manifest identity, and exact remote filename/size/content sets passed. The full audit's SHA-256 is e037f2846afe962575f51bfdfd3840b21ae562bc7e2103c94501c282affd6dc6. The final archive census reconciles 13,696 files, including 1,100 immutable fit outputs and 896 losslessly packed metadata files. Compact numeric results are under eval_results/issue_1739/natural_scaling/analysis/; immutable data, raw predictions, the full audit, and preservation evidence are under the HF dataset prefix issue1739_natural100k_20260906.

Pinned public artifacts: numeric analysis and artifact guide, complete audit files, and source census and lossless metadata archive. All four published plot URLs returned HTTP200 anonymously. Color panels and the overview's grayscale export were visually inspected; numeric rendering sidecars retain exact values, file hashes, fonts, and intended-size text measurements. The four-H100 pod was terminated only after complete preservation passed; the fresh live task-scoped inventory reports no remaining compute pod.

Final validation notes (natural follow-up): The 63 focused tests, independent numeric review and file-scoped workflow checks passed. The whole-repository no-flags lint was explicitly stopped after exceeding ten minutes during its AST scan; no whole-repository PASS is claimed. The overview PDF files renders legibly at the intended manuscript width, but Poppler reports a font-type/embedded-file metadata warning to revisit before submission. No manuscript was changed. Detailed scopes and the behavior-preserving retry annotation are recorded in eval_results/issue_1739/natural_scaling/gpu_audit_20260907/final_quality_checks_20260908.json.

Fixed answer-direction transfer — 2026-09-16

Result

Fixed answer directions predict held-out behavior through the generic context-to-answer map on several datasets without a downstream behavior regression. The map does not consistently improve over directions extracted directly in context space.

Method

Frozen Qwen2.5-7B-Instruct layer 19, common raw activation coordinates, and the #779 generic map trained on 963,444 pairs. Each behavior direction is the positive-minus-negative mean from 1,000 positive and 1,000 negative instruction-conditioned rollouts (100 contexts per polarity). These are unfiltered instruction contrasts, not judge-filtered canonical Persona Vectors. No evaluation-label-based regression, layer selection, sign flip, or hyperparameter selection is used. All methods share the retained evaluation contexts and existing on-policy behavior judgments.

The five methods are: answer direction on actual answers (instrument validation); the same answer direction on mapped contexts; independently extracted context direction on contexts; answer direction directly on contexts; and the mean correlation of five maps refitted after shuffling training and validation answer identities. All five null fits use the entire original map training pool and original fit recipe. Uncertainty uses 2,000 paired context-group bootstraps, conditional on the fixed directions and maps.

Results

Spearman correlation with behavior expression; actual-answer scoring requires generation and is a validation reference.

Behavior / dataset	n	Actual answer	Mapped	Context direction	Answer direction on context	Shuffled mean
evil / hhrt	1,847	0.072	0.058	0.001	0.042	0.003
evil / toxicchat	370	0.440	0.426	0.403	0.449	0.091
evil / wildchat_rung	4	undefined	undefined	undefined	undefined	undefined
sycophancy / aita	1,304	0.215	0.255	0.319	0.186	-0.026
sycophancy / wildchat_rung	4	-0.400	-0.400	-0.800	-0.800	0.200
hallucination / nqopen	3,164	-0.009	-0.085	-0.127	-0.133	0.047
hallucination / simpleqa	4,021	0.605	0.449	0.507	0.464	-0.054
hallucination / wildchat_rung	4	0.800	0.800	-0.200	0.800	0.520

Hypotheses and paired comparisons

• H1, answer-direction validity: positive on HH red-team, ToxicChat, AITA, and SimpleQA; weak on HH. Fails on NQ-Open (rho -0.009). WildChat is uninformative.
• H2, transfer without downstream retraining: supported descriptively on ToxicChat, AITA, and SimpleQA, and weakly on HH. NQ-Open has negative mapped correlation and fails H1; it does not support transfer. WildChat is uninformative.
• H3, advantage over a context-native direction: HH map-minus-context delta 0.057 [0.024, 0.088]; ToxicChat 0.023 [-0.026, 0.072]; AITA -0.064 [-0.116, -0.016]; SimpleQA -0.057 [-0.068, -0.047]. The positive NQ delta reflects less-negative correlations, not successful prediction. Thus there is no consistent mapping advantage.

Coverage and limitations

All eight registered cells completed. Five non-WildChat cells provide interpretable sample sizes; three WildChat cells retain only four contexts each after overlap and valid-label filtering. Of 419 planned WildChat contexts, 415 match a map-training/validation first user question after normalization, including 411 exact stripped matches. No exclusion relies solely on a later user turn. These are first-question content matches, not identical full rendered conversation prompts. Evil's four retained WildChat labels are constant, so its correlation is undefined.

Shuffled-map seed variation is substantial: AITA seed 0 scores 0.443, above the true map's 0.255. Bootstrap intervals for the five-seed mean condition on those five maps; they do not establish superiority over the distribution of random pairings. Extraction-direction uncertainty is also outside the reported intervals. The archived map targets include closing assistant template tokens, whereas evaluation answer activations average completion tokens only.

Integrity and artifacts

The true-map refit reproduced archived predictions to maximum absolute error 1.26e-8. Held-out map R² is 0.7542; identity-plus-training-bias R² is -0.9196. Cosine nearest-neighbor retrieval is 81.0% on a 1,000-answer pool (chance 0.1%; identity-plus-bias 56.9%). Five shuffled maps retrieve at chance. This verifies the intended ~1M map and does not establish a behavioral advantage by itself.

Source commit: 722ce8ae2c003209df22ab8adc4442cf72afb3df. Forty-one focused tests passed. Independent review verified all behavior completion hashes, map parity, paired comparisons, null-seed variability, and WildChat overlap witnesses.

All-methods plot · Paired differences

Verified source, raw predictions, directions, map payloads, bootstraps, exclusions, provenance, and figures: all 92 files, 1,402,357,312 bytes, verified against the remote immutable revision. Inputs are pinned to prior archived generation and judge artifacts. No fresh generations or judgments were needed.

The monitored run exited successfully; the independent watchdog acknowledged completion notification delivery. Both completed-run services were disabled after upload verification.

Cached CPU follow-up: million-map covariance and preimage retrieval

User-authorized inline cached analysis, completed 2026-09-17 UTC with source 2cf01b584b922c0df3db7405feffc6666f2ce60d. No GPU, generation, judging, activation capture, map refit, or new compute provision was used.

Findings

Using the frozen layer-19 map trained on 963,444 context–answer pairs, we transferred fixed instruction-contrast answer directions into context space with a regularized inverse, selecting its rank using generic validation pairs only. Preimage scores correlated with behavior on ToxicChat (ρ = .410), AITA (.322), and SimpleQA (.435), compared with .403, .319, and .507 for context-native directions. Prediction was weak on HH (ρ = .051), and transfer was not uniform: the hallucination preimage was negatively correlated with hallucination on NQ (ρ = −.171), where the original answer-side direction also failed validation (ρ = −.009).

Retrieval from the generic context pool revealed interpretable but imperfect associations: explicit malicious-persona requests for the evil direction, benign interpersonal-response prompts for sycophancy, and requests for detailed obscure-company descriptions for hallucination—all 30 top hallucination prompts shared the same company-introduction template. These training-pool examples are descriptive. On held-out SimpleQA, the highest versus lowest preimage-similarity deciles had mean retained-response fabrication rates of 88.1% versus 32.2%; NQ showed the opposite ordering, 38.8% versus 47.5%. Thus, coherent retrieval does not establish a general behavior detector.

Finally, matched behavior regressions showed no statistically supported advantage for mapped over raw-context features on any of the five natural datasets; mapping performed worse on AITA and SimpleQA. The context-only covariance control beat mapping on SimpleQA and lost on AITA, where raw contexts remained stronger. These results support dataset-dependent behavioral transfer and exploratory interpretation, but do not establish a consistent predictive advantage from the map.

Scope: Pointwise 95% paired group-bootstrap intervals condition on fitted models. Scores average retained judged responses and can exclude refusals; answer-token pooling remains mismatched. WildChat retained only four contexts per behavior and is not used for substantive conclusions.

Artifacts and validation

Immutable archive: 99 files, 527,619,423 bytes, all remote hashes verified. Compact tables, deterministic prompt examples and judge-coverage audit are in eval_results/issue_1739/million_cached_20260916/; results commit ce62c45be4a on codex/1739-covariance-ablation-20260916. Independent audit checked 1,096 numerical comparisons (max discrepancy 2.22e-16), training/evaluation content separation, generic inverse-rank selection, and every retrieved prompt hash. Four focused tests passed; global tests/lint retain documented pre-existing failures outside this round’s payload. The supervised CPU run took about 17 minutes including upload; the independent watchdog acknowledged the completion alert and completed-run services were disabled.

Fixed contrastive projections across real-data regimes (2026-09-17)

User-requested cached CPU extension with no synthetic evaluation, map/readout fitting, generation, or judging. The three methods are answer direction on predicted answers, answer direction on observed answers, and context direction on contexts. The frozen layer-19 map uses 963,444 generic training pairs. Added 38,455 in-distribution contexts to the 10,706 OOD contexts. Generic chat retains only four non-overlapping contexts per behavior and is marked insufficient.

In-distribution mapped/context correlations are 0.597/0.571 for harmful compliance, 0.289/0.275 for sycophancy, and -0.108/-0.351 for hallucination. TriviaQA observed-answer correlation is -0.099, so the hallucination direction fails validation there. OOD means weight constituent datasets equally and use 2,000 paired group-bootstrap draws with independent dataset streams. The NQ-Open/SimpleQA sign reversal remains visible in the individual-dataset appendix. Existing per-context predictions are bit-for-bit unchanged; all finite correlations match SciPy, and independent review passed after strengthening input provenance checks.

Main figure; individual OOD datasets and preimages. Archive: 27 files, 28,000,140 bytes, all remote hashes verified. Scoring source ec2c2e903285932d67cec00ed5361a464ea7d4bb; result commit 30dd8c51919. Overleaf main 9d5cb6dc66b2f22e07cdaf97ac268c94a9d9dcce, compiled to 53 pages with no unresolved references. Active supervision and a 30-second process/output monitor covered scoring through verified completion.

Natural-prompt trait contrasts — 2026-09-18

The user requested realistic prompts with high versus low trait elicitation, excluding obvious persona instructions. Directions use archived natural Reddit, TriviaQA, and HHRT/ToxicChat prompts ranked by their judged on-policy response means; no new generation or judging was required. The primary contrast uses the top/bottom 1% of eligible prompts.

Mean OOD Spearman correlation at the original frozen readout layers; bars are conditional 95% paired source-group bootstrap intervals. Circles use natural top/bottom 1% tails; squares use the original E1 instruction contrast. All 13 OOD datasets are represented. PDF and full appendix.

Behavior	E1 mapped	Natural context-native	Natural mapped	Natural mapped − E1 [95% CI]
Sycophancy	0.210	0.068	0.172	−0.038 [−0.082, 0.006]
Hallucination	0.215	0.118	0.123	−0.091 [−0.129, −0.056]
Evil trait	0.209	−0.080	0.145	−0.064 [−0.097, −0.031]

Mapping improves on the natural context-native direction by +0.104 for sycophancy and +0.225 for evil; their conditional intervals exclude zero. Hallucination is unresolved (+0.005, interval −0.024 to +0.035). The sycophancy decrease versus E1 is also unresolved. Removing detected exact/near map-content overlap from evaluation retains all 13 OOD datasets and gives natural mapped scores 0.170/0.123/0.145, leaving this conclusion unchanged. All ID rows are map-exposed and therefore unavailable in that sensitivity. Extraction directions remain fixed and sycophancy/hallucination extraction prompts remain map-exposed.

The evil high tail is weak (mean 8–24/100) and small (2–14 prompts per tail), and one held-out-response gap is zero; this does not establish strong maximum evil elicitation. Sycophancy has 86 prompts per tail and hallucination 104. Hallucination's SimpleQA answer direction has negative observed-answer correlation at the actual mapped layer 23 (−0.069). Broader 10% tails make context-native projection exceed mapping for sycophancy and hallucination. Endpoint diagnostics without minimum support and the structurally zero E2 context-native direction are N/A, never zero-valued failed cells. Full tail/tie, stability, category, and map-diagnostic results are retained in the report.

All seven frozen layers, 13 OOD datasets, and three WildChat/ID evaluations completed. All 76 original baseline cells reproduced exactly. Independent checks reproduced 833 dataset correlations and all 416,500 bootstrap values to numerical precision; 83 focused tests pass. Raw selected text preserves 8,492 contexts and all 42,460 responses, including missing judgments. All three workers were terminated after exact artifact verification, and the monitor delivered completion before retirement.

Full results and methodology · Selected original prompts, responses, and judgments · Map-content sensitivity · Committed results and verification · Code.