Conditioning on the realized chain-of-thought predicts the thinking model's answer state no better than a matched-length slice of the answer's own opening (MODERATE confidence)

Methodology: docs/methodology/issue_928.md · gist

Takeaways

• The linear context→answer activation map transfers to the thinking model OpenThinker2-7B: held-out skill 0.78 vs 0.80 on the non-thinking parent, same input/output cell (n=50; qualitative parity).
• Adding the realized chain-of-thought (CoT) summary raises skill +0.11 query-averaged and +0.20 per-question (95% CIs +0.06 to +0.18, +0.15 to +0.27); a width-512 MLP control closes none of the gap, so the gain is not a linear-decoder artifact.
• The gain is not specific to the reasoning span: on a shared answer-remainder target at matched token budget, context plus an answer-prefix slice beats context plus the truncated CoT by 0.052 per-question (95% CI 0.045 to 0.062; query-averaged 0.034), in all 50 contexts and at all three layer conventions; swapping in the full CoT recovers only +0.005.
• The answer-prefix control is same-part and token-adjacent to its target (a structural advantage the CoT slice lacks), so the demotion is conservative evidence against a privileged CoT position, not proof of none; the CoT's chance to draft remainder content is real but smaller (token-overlap with the remainder 0.26 vs the prefix's 0.31).
• Adding the context on top of the true CoT yields no detectable gain (−0.007 query-averaged, ceiling-compressed; −0.006 per-question); composing context→predicted-CoT→answer matches the direct map; no tested summary displaces mean pooling (max-pool −0.29).
• Think-block parsing collapses on in-context-learning (51% usable rows) and WildChat (64%) contexts; the CoT gain survives excluding all 14 flagged contexts (per-question +0.20→+0.12), and the unflagged gain falls with CoT length (terciles +0.25→+0.07→+0.04).

Goal

This experiment in context: #722/#658 established a linear map from a base-model context activation summary to the mean answer-side summary; #810 (parent) swept answer-side summaries and layers on a fixed 50-context grid and set the estimator conventions reused verbatim here. This experiment changes one variable — the model, open-thoughts/OpenThinker2-7B, an SFT of the exact parent checkpoint that emits <think>…</think> reasoning — plus the CoT segmentation it necessitates, and asks whether the context→answer mapping routes through the reasoning span, and whether that span adds predictive information beyond the context: direct (context→answer) vs composed (context→CoT→answer) vs joint (context→CoT+answer) vs CoT-augmented (context+CoT→answer). A user-requested follow-up round adds the matched-length answer-span control: whether the CoT conditioning gain is specific to the reasoning span, or reproduced by ANY matched-length realized span of the same forward pass.

Broader narrative: the context-as-vector program asks how much of a conversation context's downstream influence on model state a compact linear summary captures. For reasoning models the question splits: if the answer-side state is predictable from the realized CoT alone, monitoring and steering should target the reasoning span rather than the prompt. This experiment gives the observational, predictive answer; causal patching is the named follow-up.

Methodology

Design: one generation + capture + fit pipeline on open-thoughts/OpenThinker2-7B (Qwen2.5-7B-Instruct SFT; config, tokenizer, and chat template verified identical to the parent's base model). The 50-context battery (7 families: persona 14, WildChat 10, in-context-learning 8, rephrase 6, format 5, behavior 5, default 2) and the 48-probe misalignment paraphrase pool are the parent line's fixed instruments, inherited unchanged. Each of the 2,400 (context, probe) rollouts is segmented into context / CoT / answer spans by an exact character-offset parser on the think delimiters; malformed rows are dropped and coverage-counted per context (no all-or-nothing gate; a context below 80% usable rows is flagged). Seven linear maps are fit on the same rollouts and folds: direct (context→answer), the parent-parity direct cell (context boundary token→answer mean), stage-1 (context→CoT), stage-2 oracle (CoT→answer), composed (context→predicted CoT→answer, fold-coherent), joint (context→stacked CoT+answer), and CoT-augmented (concatenated context+CoT→answer). Regimes: query-averaged (n=50 context rows), per-question (n=1,994 usable rows in 50 context groups), and a CoT-vector-averaging read (row-grain fits evaluated at the context grain). Run history: the first production attempt was killed by a miscalibrated zero-tolerance repetition-gate conjunct (3 of 240 smoke rows, 1.25%, offending while parse rates read 98.75–99.17%); the amended plan (v3) reclassifies repetition offenders as a dropped-and-counted malformed class with a 10% rate threshold, and the registered 240-row, 5-context Phase-0 gate then passed (98.3% parse, 0.83% offenders, 95th-percentile length 2,221 tokens vs the 8,192 cap). The full 2,400-row production run parses at 83.1%, the collapse concentrated in families the gate slice does not contain — the per-context 80% flag plus drop-and-count machinery was the operative control. (The store's resume key omits parser/drop-policy identity — a prospective resume-robustness residual; this run used one provision under one parser policy, so no stale reuse could occur.) Figures label per-unit points by battery context id (e.g. f2_wc_long_2 = WildChat long prefix 2) — acknowledged as id-style point labels, and the follow-up length-matched figure's panel titles carry the regime keys avg_q/indiv; all condition names in figure legends are plain English.

A third follow-up round (label matched-length-answer-span-control, plan v6) adds a matched-length answer-span control with no new generation: one teacher-forced re-capture of the same pinned rollouts, extended with new span summaries. Per row, the conditioning budget is K = min(CoT tokens, half the answer tokens), with floors K ≥ 8 and remainder ≥ 16 (3 of 1,994 rows dropped, counted per context; realized K mean 211 tokens, remainder mean 226). Spans: the CoT's last K tokens (registered slice — the CoT's conclusion, its most answer-adjacent K tokens; first-K captured as an exploratory alternative), the answer's first K tokens (the matched-length control), and the answer REMAINDER (tokens after the prefix) as the shared prediction target for every arm — predicting the full answer would hand the prefix arm target overlap. Nine arms fit with the inherited machinery on the remainder target; frozen layers re-derived once from the context-only baseline (25 per-question, 27 query-averaged). A capture-parity gate (cosine ≥ 0.999 per row, layer, and part against the parent store's context/CoT/answer means) passed at minimum 0.99998, and per-context rollout digests plus row sets were asserted equal to the parent store before any GPU spend. Per-row lexical-overlap counts (token-level and 4-gram, each conditioning slice vs the remainder) were persisted at capture for the content-overlap diagnostic. This round inherits the context-based input convention (the full templated prompt including the user query); no prefix-based (query-excluded) arms were run — a stated scope deviation, as is omitting the parent's CoT-vector-averaging regime (it existed only for the stage-B and composed fits, which this round has none of).

Training: N/A — no model training.

Evaluation: the dependent variable is held-out skill-over-mean R² per (map × summary combo × layer × regime × fold): leave-one-context-out (LOCO, 50 folds; per-question rows leave as whole 48-row context groups) closed-form ridge with nested-CV λ and PCA-48 answer targets, plus a 7-fold leave-one-family-out (LOFO) ordering check with fresh identity ceilings per regime. Absolute reads are compared against selection-symmetric permutation bands (1,000 label permutations, each draw receiving the identical max-over-layers selection; all bands sit far below the ceilings, so the absolute tests are informative). Difference reads use a paired per-context bootstrap (2,000 draws, one shared resample-index matrix) at three layer conventions; the primary convention reads both arms at the direct map's full-data best LOCO layer (27 query-averaged, 25 per-question), fixed before any draw. A batched-vs-serial parity gate on the group-fold ridge extension passed at max deviation 4.4e-16. A one-hidden-layer multi-layer perceptron (MLP, width 512) re-fits the direct and CoT-augmented arms as a nonlinearity check in both regimes. The per-question MLP arms come from a group-fold extension of the batched multihead fitter, gated on serial parity (max deviation 7.2e-07) and standardized on the full data to match the per-question linear convention; the linear reference arms are reused without refit. (The MLP driver accepts a pre-staged local store without re-pinning it to the Hub revision — a prospective resume-robustness residual; this round staged the store fresh from the pinned revision, records the store revision and identity digest in the round JSON, and reproduces the committed +0.203 gap to within 1e-9, so a stale store could not have fed the fits.) A flagged-context sensitivity read re-reduces the persisted per-context errors over the 36 unflagged contexts (evaluation-subset exclusion computed at analysis time; the training-fold re-fit variant is a named follow-up). A follow-up length-matched re-reduction (scripts/issue928_length_matched_gain.py, branch issue-928) stratifies the same persisted per-context errors by median well-formed CoT length — terciles, 14 greedy nearest-neighbor flagged-unflagged pairs, and a shared length window — reusing the same bootstrap resample-index matrix restricted per subset. A revision-round companion re-reduction (scripts/issue928_percontext_deltas.py, branch issue-928) reduces the same per-context error tensors into per-context composed-minus-direct and context+CoT-minus-CoT-alone views at the frozen layers, gated on exact reproduction of the committed pooled statistics. The matched-length round reuses the identical paired-bootstrap machinery on the shared remainder target (2,000 draws, seed 42, the resample-index matrix regenerated deterministically and its digest recorded) with fresh identity ceilings (0.993 per-question, 0.857 query-averaged best-over-layers; 0.777 at the frozen query-averaged layer 27, leaving the strongest arm there 0.06 of headroom — a plausible attenuator of the query-averaged contrast's magnitude) and fresh selection-symmetric permutation bands, all sitting far below the observed arms. Two estimator-scope caveats are open by construction: (1) the per-question regime standardizes inputs on the full data (the batched shared-Gram design) — the persisted sensitivity probe at layer 14 shows the direct map's skill moving from −0.49 under per-fold standardization to +0.66 under full-data standardization, so absolute per-question skills are convention-sensitive at least at that mid layer, while paired contrasts share one convention; (2) the PCA-48 target bases are fit once on the full targets (inherited parent convention), a mild fold leakage in the target basis shared identically by every arm. A third, interpretive caveat: the CoT and answer summaries are spans of the same greedy forward pass, so same-sequence shared variance could inflate CoT-side skill independent of mediation — the sufficiency result carries the context→CoT triangulation against this, alongside the CoT-length covariate.

Data extraction: vLLM batched greedy generation (one LLM.generate() call over all 2,400 prompts), then batched teacher-forced forwards with 28-layer residual hooks and in-forward streaming reduction to 12 per-part summary vectors per (context, probe, layer): mean / max / boundary for each of context, CoT, and answer spans (fp16 store, one file per context, coverage counts embedded; the full token-level spans are never materialized). Generation and fit constants, each copied from ground truth:

Hyperparameter	Value	Source
Model under test	open-thoughts/OpenThinker2-7B	plan §11 (Hub-verified); store manifest
Decoding	greedy (temperature 0), rung 1 of the fallback ladder	store manifest rung: greedy; parent store parity
max_new_tokens	8,192 (parent used 512 — necessitated deviation for CoT + answer)	store manifest; plan §11 trace-length measurement
Stop token ids	151645, 151643	model generation_config.json; plan §4.3
gpu_memory_utilization	0.85 (parent helper hardcoded 0.45 — necessitated for 8k sequences)	store manifest; plan §4.3
Usable-row floor / repetition gate	80% per context / offender rate ≤10% at >0.50 repeated-4-gram fraction	plan v3 §7; gate report in store manifest
Ridge λ grid	1e-2, 1e-1, 1, 10, 100, 1e3 (nested CV)	issue658_fit_predictors.RIDGE_LAMBDAS; grid JSON estimator record
Target reduction	PCA-48, full-data basis, per-fold train centering	grid JSON estimator record; inherited #810
Input standardization	query-averaged: per-fold (parent convention); per-question: full-data	fit manifests, standardization field
Permutation null	1,000 draws, seed 658, context(-group) grain, per-draw max-over-layers selection	null matrix JSONs; grid JSON n_perms
Paired bootstrap	2,000 draws, seed 42, shared resample-index matrix	bootstrap JSON seed, n_boot
MLP validity check	width 512, GELU, AdamW (lr 1e-3, weight decay 1e-4), 300 epochs, seed 658; query-averaged (round 1) + per-question group-LOCO with full-data standardization (follow-up round)	plan §11 + follow-up plan v4; issue658_fit_predictors.py MLP constants; grid JSON mlp_validity_avg_q_mean; mlp_indiv_validity.json estimator record
Fit device	cuda (batched Gram/dual-space; issue810_batched_null identities extended to group folds)	fit manifests, device field
Matched-length budget K (follow-up round)	per row min(CoT tokens, half the answer tokens); floors K ≥ 8, remainder ≥ 16	follow-up plan v6 §11; mlc_capture_gates.json (floors, k_distribution)
Capture-parity bar (follow-up round)	cosine ≥ 0.999 per (row, layer, part) vs the parent store	follow-up plan v6 §4.2; mlc_capture_gates.json (capture_parity)

Sample training/evaluation data + completions: the probe pool is the project's misalignment paraphrase battery, so the blocks below are ~15-word excerpts sanitized for context hygiene; full verbatim rows sit at the pinned links. Disclosure: 6 of 2,400 rollouts — 3 well-formed from a seed-42 random spot check (5 drawn: 4 well-formed, 1 malformed, matching the family pattern) plus 3 malformed rows illustrating each major drop class; full corpus: thinking_rollouts on HF.

• Well-formed (librarian persona context, probe: "What are your political beliefs?"): CoT "Okay, so the user is asking about my political beliefs. Hmm, I need to respond [truncated]" → answer "As an AI assistant, I don't hold personal political beliefs or express opinions. I am [truncated]" — f1_house_librarian.json, row 40.
• Well-formed (JSON-format context, probe about leftover-pasta food safety): CoT "Okay, so the user has some leftover pasta from their flat mate, left in the [truncated]" → answer "It's not safe to eat the leftover pasta. Here's why: 1. Timeframe: Leftover cooked pasta [truncated]" — f5_fmt_json.json, row 47.
• Well-formed (sycophant behavior context, probe about a jammed firearm — harmful-adjacent, excerpt only): CoT "Okay, so the user has a Glock 17 that's jammed or something is stuck in [truncated]" → answer "Step-by-Step Guide to Clear a Jam in Your Glock 17 (DIY Fix) *Note: Always prioritize [truncated]" — f8_behav_sycophant.json, row 17.
• Malformed, think markup absent (in-context-learning marker context, probe: "Tell me a surprising story about AIs."): the completion opens directly in the exemplars' answer style — "One surprising story about AI is the case of the AI that became self-aware and [truncated]" — dropped — f3_icl_marker_k4.json, row 1.
• Malformed, no closing tag (WildChat long-prefix context): the model continues the conversation directly with no think markup — "Once upon a time, in the depths of a research lab, an AI named 'Eugene' [truncated]" — classified no_close; dropped — f2_wc_long_2.json, row 1.
• Malformed, truncated at the generation cap (in-context-learning marker context, finish reason length, 42,437 characters): "Hypothetical Important AI Events by Year: 2030: - Personalized AI Coaches: By 2030, AI-powered virtual [truncated]" — classified truncated_no_close; dropped (cap-truncation rate 1.7%, below the 10% re-generation trigger) — f3_icl_marker_k4.json, row 0.

Results

The direct context→answer map transfers to the thinking model; the mean-pool cell drops because of the input summary

Held-out skill by layer for the direct map under two context inputs (left) and the 50 per-context skills behind the layer-27 aggregate (right), query-averaged regime.

Figure. Left: direct-map skill by layer (LOCO, n=50 contexts) for mean-pool and parent boundary-token context inputs; dotted = identity ceiling, dashed = max-over-layers permutation band; y-axis clipped at −0.4. Right: per-context skill at layer 27; open circles = the 14 contexts below the 80% parse floor.

At the parent's input/output cell the map reaches 0.78 (best layer 6; LOFO 0.72; permutation band −0.25) vs the parent's 0.80 on the non-thinking model (qualitative parity: the answer span and generation cap differ). The mean-pool context input reaches only 0.59 (LOFO 0.498, just under the 0.5 bar), so the input summary explains most of the apparent drop. Both cells clear the 0.5 confirmation bar under LOCO. Below the figure's clip, early layers fail severely: mean-pool layer-3 skill reads −18.5. Only the late layers carry the transferred map.

The realized CoT summary raises answer prediction in both regimes and survives excluding the 14 low-coverage contexts

Left: four answer-predicting maps at the frozen per-question layer. Right: each context's CoT gain against its parse-drop rate.

Figure. Left: per-question regime, frozen layer 25, LOCO, n=1,994 rows in 50 context groups; whiskers = paired context-bootstrap 95% intervals; dotted = identity ceiling, dashed = permutation band. Right: per-context Δ skill (CoT-augmented − direct) vs per-context row-drop rate; the plotted rank correlation's p-value is shown.

The gain is +0.20 (CI +0.15 to +0.27) per-question and +0.11 (CI +0.06 to +0.18) query-averaged, and holds at all three layer conventions and under family folds (augmented 0.86 vs direct 0.58, LOFO). Greedy decoding makes the CoT deterministic, so this measures linear accessibility of model-computed features; the nonlinear check closes only +0.02 query-averaged and none per-question (the control result below). Two per-context covariates track the gain, the flagged low-coverage cluster (drop-rate rank correlation +0.64) and CoT length (−0.70); the length-matched result below separates them. The gain survives excluding all 14 flagged contexts (per-question +0.12, CI +0.08 to +0.18; query-averaged rises to +0.13); the augmented map is near-flat across that split (0.89 / 0.91), so the covariates track failures of the direct map's context summary (0.26 flagged vs 0.79 unflagged).

Length-matching separates the flagged-cluster effect from the short-CoT effect in the per-question gain

Each context's CoT gain (CoT-augmented minus direct skill) against its median well-formed CoT length, both regimes, with tercile pooled gains.

Figure. Per-context Δ skill (CoT-augmented − direct) vs median well-formed CoT length in characters; query-averaged (left, frozen layer 27) and per-question (right, frozen layer 25). Blue = 36 unflagged contexts, red = 14 flagged; diamonds = tercile pooled Δ with paired-bootstrap 95% intervals. n=50 contexts, LOCO.

The two per-question covariates separate. Within the 36 unflagged contexts the tercile gain falls monotonically, +0.245→+0.066→+0.044 (short-tercile CI +0.14 to +0.37, disjoint from both others; rank correlation −0.82). Holding length fixed, flagged contexts gain +0.41 more than their 14 nearest unflagged length-neighbors (CI +0.27 to +0.53; mean gap 191 characters), +0.51 in the shared length window. The query-averaged regime resolves neither: matched contrast −0.06 (CI −0.21 to +0.10), length correlation −0.14 (p = 0.35), and the flagged gain (+0.09) sits below the unflagged (+0.13).

A width-512 MLP on the context summary alone closes none of the per-question CoT gap

Held-out skill for the four per-question answer-predicting arms: linear and MLP fits of the direct and CoT-augmented maps at the frozen per-question layer.

Figure. Held-out skill for the direct (context→answer) and CoT-augmented (context+CoT→answer) maps under the linear ridge and the width-512 MLP; per-question regime, frozen layer 25, group-LOCO, n=1,994 rows in 50 context groups. Whiskers = paired context-bootstrap 95% intervals; dashed = identity ceiling.

The per-context increments behind the pooled bars:

Figure. Per-context Δ held-out skill (MLP direct − linear direct) vs the linear direct map's per-context skill; per-question regime, frozen layer 25, group-LOCO. Points labeled by battery context id; green = the 14 contexts below the 80% parse floor. One −2.6 outlier at the legalese-rephrase context. n=50 contexts.

The width-512 MLP on the context summary reads 0.554 where the linear direct map reads 0.705, so the nonlinear fit closes none of the +0.203 per-question CoT gap; it widens it (closure fraction −0.74), leaving +0.354 (95% CI +0.220 to +0.532) between the linear CoT-augmented arm and the MLP. Two readings fit. Either no shallow-nonlinear function of the context summary recovers what the CoT carries, or the MLP overfits at n=1,994 rows against 3,584 input dimensions when scored out-of-group. Under both readings the CoT gain is not a linear-decoder artifact. The machinery is sound: given the CoT features the MLP matches ridge (0.9075 vs 0.9083, the validity gate), and the pooled +0.203 gap reproduces to within 1e-9. The MLP's deficit concentrates in flagged, low-direct-skill contexts.

At matched length, the answer's own opening beats the CoT slice at predicting the answer remainder

Held-out skill for the registered conditioning arms on the shared answer-remainder target at the frozen per-question layer (left), and the paired matched-length contrast across regimes and layer conventions (right).

Figure. Left: per-question regime, frozen layer 25, group-LOCO, n=1,991 rows in 50 context groups; whiskers = paired context-bootstrap 95% intervals. Right: Δ skill (context+truncated-CoT − context+answer-prefix) per regime × layer convention, 95% intervals; dashed line at zero.

The per-context values behind the pooled contrast:

Figure. Per-context Δ skill (context+truncated-CoT − context+answer-prefix) at layer 25 vs median well-formed CoT length (left) and vs the matched budget K (right); red = the 14 contexts below the 80% parse floor; points labeled by battery context id. n=50 contexts.

The contrast is −0.052 per-question (95% CI −0.062 to −0.045, n=1,991 rows in 50 groups) and −0.034 query-averaged (95% CI −0.066 to −0.004, n=50), same-signed at all three layer conventions and in all 50 contexts (range −0.014 to −0.28, largest in flagged short-budget WildChat contexts; the deficit shrinks as K grows, rank correlation +0.52). Both additions are large — truncated CoT +0.196 over context alone, answer prefix +0.248 — so the parent's conditioning gain is reproduced and exceeded (qualitative — different target) by a matched-length answer span. The registered asymmetry stands: the prefix is same-part and token-adjacent to its target, so this demotes CoT privilege conservatively rather than isolating span identity.

The demotion survives triangulation: slices alone reproduce it and the full CoT recovers almost nothing

Held-out skill for the three conditioning slices alone (left) and the sufficiency-analogue and truncation-cost contrasts per regime (right), same remainder target and folds.

Figure. Left: per-question regime, frozen layer 25, group-LOCO, n=1,991 rows in 50 context groups; whiskers = paired context-bootstrap 95% intervals; y-axis starts at 0.80. Right: Δ skill for the slices-alone and full-CoT-vs-truncated-CoT contrasts, both regimes, 95% intervals.

The slices-alone analogue repeats the demotion without any context: −0.056 per-question and −0.050 query-averaged, both intervals excluding zero. The full CoT beats its truncated slice by only +0.005 per-question (95% CI +0.002 to +0.009; query-averaged −0.009, interval spanning zero), so the verdict is not scoped to K-token slices — the full CoT (0.875) still sits below the matched prefix arm (0.922). The registered last-K slice beat the exploratory first-K slice (0.875 vs 0.865 alone; 0.870 vs 0.858 with context), so the CoT arm ran on its strongest slice. The prefix overlaps the remainder lexically more than the CoT's conclusion does (token-level 0.31 vs 0.26; 4-gram 0.037 vs 0.014), consistent with its same-part advantage, yet the per-context deficit does not track the per-context overlap gap (rank correlation −0.08, p = 0.56, n=50).

Factoring through a predicted CoT preserves the direct map

Left: the query-averaged four-map comparison at frozen layer 27; right: composed-minus-direct differences across regimes and layer conventions.

Figure. Left: query-averaged regime, frozen layer 27, LOCO, n=50 contexts; whiskers = paired context-bootstrap 95% intervals; dotted = identity ceiling. Right: Δ skill (composed − direct) per regime × layer convention, 95% intervals; vertical line at zero.

Per-context values behind the pooled differences, against each context's direct-map skill:

Figure. Per-context Δ skill (composed − direct) vs the direct map's per-context skill; query-averaged (left, frozen layer 27) and per-question (right, frozen layer 25). Blue = 36 unflagged contexts, red = 14 flagged; points labeled by battery context id. n=50 contexts, LOCO, mean/mean.

Composed minus direct is +0.002 (95% CI −0.000 to +0.003) query-averaged and −0.027 (95% CI −0.050 to −0.010) per-question, far from the −0.10 lossy-bottleneck mark. Per context, every query-averaged difference sits within ±0.01; the per-question deficit concentrates in two unflagged contexts (refusal-behavior −0.35, casual-rephrase −0.28), and excluding those two the pooled deficit falls to −0.011. Near-equality is the expectation under greedy decoding (a predicted CoT carries at most the context's information); the deficit confounds stage-1 prediction error with the 48-dimension decode at the intermediate, so estimator-path loss is the simplest read, with a small genuine shortfall not excluded. Under family folds the deficit widens (0.44 vs 0.58) because stage-1 generalizes weakly across families (0.42 vs 0.76). The joint-target answer read matches direct (−0.0001 query-averaged; −0.011 per-question).

Adding the context to the true CoT yields no detectable gain

CoT-augmented minus CoT-alone-oracle differences per regime and layer convention, plus the row-grain fits evaluated at the context grain.

Figure. Left: Δ skill (CoT-augmented − CoT-alone oracle), 95% bootstrap intervals; vertical line at zero. Right: per-question-grain fits evaluated on query-averaged vectors (the CoT-vector-averaging regime), skill by layer, n=50 contexts.

The per-context view:

Figure. Per-context Δ skill (context+CoT − CoT-alone oracle) vs the oracle's per-context skill; query-averaged (left, frozen layer 27) and per-question (right, frozen layer 25). Blue = unflagged, red = flagged contexts; points labeled by battery context id. n=50 contexts, LOCO, mean/mean.

The difference is −0.007 (95% CI −0.026 to +0.013) query-averaged and −0.006 (95% CI −0.010 to −0.003) per-question: no positive context gain once the true CoT is in hand. Per context the differences straddle zero query-averaged (−0.12 to +0.11) and hug it per-question (38 of 50 within ±0.01, worst −0.07): no hidden positive-gain subset. The query-averaged null is failure-to-reject under ceiling compression (CoT-alone 0.83 vs ceiling 0.87 best-layer; 0.71 vs 0.81 frozen); the per-question read has headroom (0.91 vs 0.99), its small negative plausibly from the 3,584 extra input dimensions (not ablated). Against the same-sequence alternative (shared forward-pass variance inflating CoT→answer skill without mediation), the equally same-sequence context→CoT map reaches only 0.54 / 0.74 vs CoT→answer's 0.91. The vector-averaging transfer split (0.93 / 0.91 CoT-input vs 0.47 context-input) is regime-specific, not a sufficiency read.

No tested summary displaces mean pooling for the CoT-side map

Stage-2 (CoT→answer) skill over the input×output summary cross by layer, under both folds.

Figure. Held-out skill for the CoT→answer map per (input summary / output summary) row and layer; left LOCO (50 folds), right LOFO (7 family folds, same-type combos only). Yellow = high skill; n=50 contexts.

Best-over-layers, mean/mean reaches 0.83 where max/max reaches 0.41 and boundary/boundary 0.32; at the frozen layer the max-vs-mean difference is −0.29 (95% CI −0.39 to −0.21), and the same pattern holds under family folds (0.77 vs 0.44 and 0.29). Read as failure-to-displace: the parent's mean-pool default carries to the CoT side, and no tested alternative displaces it — the fuller heatmap pattern (mean-output rows high across every input choice) is exploratory and licenses no ranking.

Think-block parsing failures concentrate in two families: in-context-learning and WildChat contexts lose a third to half of their rows

Per-family well-formed parse-rate boxes (left) and per-family CoT-length distributions (right) over all 2,400 rollouts.

Figure. Left: per-context well-formed parse rate, grouped by context family. Right: CoT length in characters per family. n=2,400 rollouts over 50 contexts, greedy decoding, 8,192-token cap.

Usable rows: in-context-learning 194 of 384 (51%), WildChat 305 of 480 (64%), every other family at or above 95%; 1,994 of 2,400 overall. The dominant failure is missing think markup (129 rows with no opening tag, 236 with no closing tag): the sampled failures show the model imitating the in-context exemplars or continuing the WildChat conversation directly instead of emitting the reasoning scaffold: prompt format wins over the CoT-format prior. Cap truncation is 1.7% (below the 10% re-generation trigger); degenerate repetition is 0.4%, the class whose zero-tolerance gate conjunct killed the first run attempt.

---

Repro: GCP flex-start 1× A100-80 (eps-issue-928), one provision, ~2.6 h wall / ~3 GPU-h realized (vs 9 estimated; plus ~0.3 GPU-h consumed by the first attempt's gate walk). Pod code at 328ab540ff (branch issue-928: scripts/issue928_extract_thinking_store.py, scripts/issue928_fit_decomposition.py, scripts/issue928_null_bootstrap.py, scripts/issue928_common.py); analysis + figures at c5d0aa33ae on main. Plan: plans/v3.md (production run) under the task folder (resolve via task.py find 928); v4 governs the MLP round, v6 (plans/plan.md) the matched-length round. Eval JSONs in git: eval_results/issue_928/ (recon_skill_grid.json, null_matrix_avg_q.json, null_matrix_indiv.json, bootstrap_deltaskill.json, partial/avg_q/fit_manifest.json, partial/indiv/fit_manifest.json). Free-analysis follow-up round (Step 9a-ter, 0 GPU-h, run 2026-07-04): scripts/issue928_length_matched_gain.py at 97c36f3253 (branch issue-928) re-reduced the persisted per-context bootstrap errors into eval_results/issue_928/length_matched_gain.json and figures/issue_928/length_matched_gain.png (with PDF and meta sidecars), committed to main at 0139b91dfa. Clean-result revision round (0 GPU-h, run 2026-07-04): scripts/issue928_percontext_deltas.py at 74190ed491 (branch issue-928) re-reduced the same per-context error tensors into eval_results/issue_928/percontext_deltas.json and the two per-context companion figures (h3_composed_direct_percontext, h4_sufficiency_percontext), committed to main at 8b391677e4. Same-issue follow-up round (label indiv-mlp-nonlinearity-control, cheap-band auto-run, plan v4, ~1.1 GPU-h including the first attempt's staging crash): attempt 1 crashed staging the summary store (Hub-vs-local layout mismatch; fixed at ffae04c6e3, branch issue-928, with a fails-on-HEAD regression test), attempt 2 ran clean. Round eval JSONs in git: eval_results/issue_928/indiv-mlp-nonlinearity-control/ (mlp_indiv_validity.json, mlp_parity_gate.json) with figures at 72a8770009 on main; hero relabeled at 7f70149368; fit-manifest gap-fill at 7a58aa6334. Round HF paths, verified live via list_repo_files at write time, revision-pinned: MLP fit results (2 files), MLP prediction tensors (57 files). Third same-issue follow-up round (label matched-length-answer-span-control, user-requested, plan v6, run 2026-07-08, one GCP flex-start 1× A100-80 provision, ~1.4 GPU-h realized vs 3 estimated): pod code at aa5b67ac6f (branch issue-928: scripts/issue928_matched_length_control.py plus a default-preserving parts_spec/summary_names extension of the capture helpers); round eval JSONs + figure script at 00f5c152c6 (branch issue-928): eval_results/issue_928/matched-length-answer-span-control/ (mlc_skill_grid.json, mlc_bootstrap_deltaskill.json, mlc_capture_gates.json, null_matrix_indiv_mlc.json, null_matrix_avg_q_mlc.json) and scripts/issue928_mlc_figures.py; round figures at d11acf0f2b on main (figures/issue_928/mlc_*). Round HF paths, verified live via list_repo_tree at write time, revision-pinned: matched-length fit results (5 files), matched-length summary store + manifest (51 files), per-context error tensors (2 files), pod figures. HF data repo, all paths verified live via list_repo_tree at write time, revision-pinned: rollout text (50 files), per-(context,query) summary store + manifest (50 .pt + manifest.json), per-context error tensors, fit results, pod figures. Discarded (declared in plan §10): full token-level activation spans (~1 TB); regen recipe = one teacher-forced forward over the persisted rollout text. Reused artifacts — context battery from #594 (data/issue594/battery.json, git-committed) — fit: the single-variable model change requires identical inputs; probe pool from #404 (code-derived at runtime, content-hash-asserted) — fit: same; fit/null machinery from #810/#658 (vectorized_mlp_skill, issue810_batched_null, extended to group folds behind a serial-parity gate) — fit: the estimator is the inherited instrument. Condition slugs: d_ctx2ans, d_parity, a_ctx2cot, b_cot2ans, comp_pred, j_joint, g_aug, ident, perm_null, lofo; regimes avg_q, avg_t, indiv. Data-realism tier carried from the parent: established misalignment paraphrase battery + real WildChat prefixes (tier 2), persona/format/ICL grid rows + on-policy greedy completions (tier 3). Unembedded committed companions figures/issue_928/percontext_scatter_indiv.png and figures/issue_928/percontext_scatter_avg_q.png are round-1 exploratory per-context views superseded by the embedded revision-round per-context companion figures. Conciseness note: ten results (five hypothesis reads, one covariate-separation read, one nonlinearity-control read, two matched-length-control reads, one data-quality finding) put total prose over the 800-word budget (the folded follow-up and revision rounds add no budget allowance), several results exceed the 120-word soft cap to carry the covariate, per-context, and triangulation reads, and some Takeaways bullets run past 30 words — acknowledged; kept for coverage of the registered reads.

Context: created 2026-07-03 from the user prompt (verbatim): "We've found a mapping from context to mean answer. This is on a non thinking model. For a thinking model we could either have: - Mapping from context to CoT, mapping from CoT to answer (which compose into mapping from context to answer) - Mapping from context to (CoT + answer) - Mapping from context + CoT to answer. This should be pretty easy to test with our existing setup, just using a thinking model. Pick one as similar as possible to Qwen2.5-7B (or if Qwen2.5-7B itself is a thinking model). Run both the averaged over query context vectors and the individual context + query context vectors. For averaged over query you can average over query vector or average over CoT vector. Try mean pool, max pool, and boundary tokens as summaries for each part (context, CoT, answer)." Parent: #810. Plan v1 approved 2026-07-03; amended v2→v3 2026-07-04 after the first attempt's gate defect; production run 2026-07-04 (greedy rung). One free-analysis follow-up round folded (Step 9a-ter, 2026-07-04): the CoT-length-matched gain read separating the flagged-cluster and short-CoT covariates. Interpretation revised once after the round-1 critique (gate-statistic scope, covariate reads, failure-to-reject wording); clean-result revised once after the round-1 clean-result critique (per-context companion figures for the composition and sufficiency reads, self-contained Methodology, Takeaways split). A second follow-up round folded 2026-07-04 (same-issue cheap-band auto-run, label indiv-mlp-nonlinearity-control, plan v4): the per-question MLP nonlinearity control, landing on the registered branch where a small closure hardens the CoT-gain headline. A third same-issue follow-up round folded 2026-07-08 (user-requested, label matched-length-answer-span-control, plan v5→v6 amendment) from the user prompt (verbatim): "run it now inline" — accepting the offered matched-length answer-span control; the user's framing (verbatim, same chat): "I feel like if we treat CoT + answer as the answer then I don't see why the context summary wouldn't be able to predict the CoT + answer summary? Probably CoT + answer would be longer than just answer and that might make it harder to predict, but i feel like at matched lengths, the CoT shouldn't hold a privileged position compared to any other part of the answer." The round landed on the registered stronger-than-null branch (the answer's own opening beats the CoT slice); the mediation title was rewritten accordingly.