Linear context-to-answer mapping quality does not track capability across a fixed-size release column: no-thinking reads invert on the generic corpus (MODERATE confidence)

Takeaways

• 0.788 → 0.719 → 0.734: calibrated retrieval acc@1 across the fixed-size 27B release column with thinking off; the newest release sits 5.4 points below the oldest (n = 987).
• 0.784, 0.784, 0.781: the thinking-read column is flat, every contrast interval straddling zero, and complete-case; row drops reach 14.1% of test prompts in the newest cell.
• 0.569 → 0.597: identity-plus-bias retrieval rises while the learned-map increment shrinks (0.221 to 0.139); the deficit is generic-surface only, and residualization shrinks the endpoint gap to 0.5 points.
• 6 of 7: thinking checkpoints gain on the generic corpus from end-of-thought reads (+0.03 to +0.11); hard-set gains are inconsistent and not larger (4 of 7 positive, p = 0.109).
• rho = 0.05 and 0.23 (n = 9 each, critical about 0.68): no capability trend in either read; within-family scaling is monotone only with thinking off (0.628 to 0.788).
• At most 0.001: endpoint movement under language-intrusion recounts, banked cells included; the at-chance cell persists truncation-free; precision stays conditional on one frozen rollout, frozen layers, and a reconstructed hard set.

Goal

• This experiment in context: The context-to-answer mapping line fits a linear map from a model's pre-answer residual state to its answer representation and scores it by held-out retrieval. This experiment asks whether that mapping quality is a capability correlate: does it rise with the Artificial Analysis intelligence index across a cross-family panel, and within a fixed-size 27B release column (Qwen3.5, Qwen3.6, Qwen3.8) where width, depth, and family are constant? The prior issue in this line, #2330, reported a Qwen2.5-7B anchor beating Qwen3.5-9B on held-out R² (0.705 vs 0.661, p = 0.002) at a fixed depth-matched layer; this run reads 0.687 vs 0.664 at validation-frozen sweep-winner layers, not directly comparable, but the direction replicates, and the two are near-equal on the retrieval primary (0.731 vs 0.726 calibrated).
• Broader narrative: Whether linear context-to-answer structure is a general-capability correlate or a model-specific property decides how the map can be used as a probe. The answer here supports the model-specific reading within this panel: newer or more capable checkpoints do not have more linearly decodable answer states.

I acknowledge the per-result prose-band and total-prose-budget WARNs: every quoted rate keeps its n, chance, and interval status inline because those denominators condition the interpretation.

Methodology

Design: 12 open-weight checkpoints (Qwen3.5 at 0.8B/2B/4B/9B/27B; Qwen3.6-27B; Qwen3.8-27B; OLMo-3-7B Instruct and Think; OLMo-3.1-32B Instruct and Think; a Qwen2.5-7B anchor) under two chain-of-thought settings give 21 fitted maps. The manipulated variable is the checkpoint; the fixed-size 27B release column is the primary comparison, and the ordered-release criterion asks whether the newest release beats the oldest with both adjacent steps consistent.

With thinking off the map input is the residual state at the last prompt token; with thinking on it is the state at the generated end-of-thought boundary token. OLMo Think checkpoints additionally capture a prompt-side pre-think state, decomposing reasoning training from read position.

Surfaces: the generic real-user corpus (train 10,000 / validation 400 / test 1,000 prompts) and GPQA Diamond (198 questions, 5 rollouts each) as a transfer surface that is never fitted. One rollout per generic prompt at seed 42; all models see identical rendered rows. A zero-GPU follow-up analysis round (2026-08-27, driver scripts/issue2588_followup_9ater.py) added a truncation-free recompute of the newest-27B thinking hard-set read, closed the banked-cell language-intrusion audit, and recorded the layer re-selection bootstrap as GPU-bound; no new generations or fits.

Training: N/A — no model training.

Evaluation: Two surfaces probe the construct (how well the pre-answer state linearly determines the answer representation): the generic real-user corpus, fitted and scored on held-out prompts, and GPQA Diamond, a never-fitted transfer surface, because the Goal asks whether mapping quality tracks capability on ordinary text and on questions small models cannot answer. Preprocessing: identical rendered rows, one seed-42 option shuffle per hard-set question, sha-pinned splits, per-cell drop accounting. Per map, a ridge regression maps the input state to the answer vector (token-mean over answer tokens; thinking rows use the post-think segment only), scored by held-out retrieval. The table fixes each read and its uncertainty recipe.

Read	Recipe
Primary retrieval	held-out kNN acc@1 by cosine among the 1,000-row test pool (chance 0.001), calibrated: raw minus the cell's shuffled-pairing null mean (200 permutation draws, per-draw lambda re-selection)
Baselines	identity-plus-learned-bias (R² and retrieval form), held-out R² as a within-model diagnostic, length-only, train-fold length residualization
Repeat-draw reference	seed-43 answer vectors retrieving seed-44 targets; a repeatability comparator, not an attainable bound (five end-of-thought maps exceed it, up to 1.179 normalized)
Layer choice	frozen once per map on the validation split from a dense sweep
Column contrasts	paired bootstrap over shared test prompts (1,000 draws, seed 42, one shared resample matrix), shifted by the null-mean difference; intervals conditional on the frozen layers and rollouts
Hard-set transfer	same-question retrieval acc@1 among the 990-row pool (chance 5/990), the model's own answer as target, correct or not; intervals resample whole question clusters (1,000 draws, seed 42) because a question's five rollouts are correlated
Behavioral accuracy	anchored exact match on the answer letter; fallback judge claude-sonnet-4-5-20250929 (Batch API, extraction-only rubric: system prompt + user template at the run SHA) fired on 5 cells: 430 calls, 0 malformed returns, 4 rows dropped on transport errors, residual unparseable at most 0.3% per cell; per-cell verdicts persisted as fits/<cell>/gpqa_judge_verdicts.json at issue2588_capability_panel/fits @ b8f456d49f
Language-intrusion audit	per-cell CJK scan of every Qwen pool conditioned on prompt language, banked cells included (closed by the follow-up round): 8 to 36 intruded rows per cell (0.85% to 4.20% of non-CJK-prompt rows); excluding them moves the endpoint contrast from -0.0537 to -0.0536 (n = 970); zeroing moves levels 0.71 to 3.52 points, swapping one near-tied thinking pair whose interval straddles zero, so no verdict flips; banked cells read raw/zeroed/excluded 0.732/0.705/0.730 and 0.727/0.705/0.722 (34 and 24 intruded of 955 eligible; derived counts match the source issue's own scan exactly), max movement 0.027, same class as the fresh cells

Constant	Value	Source
Decoding	temperature 1.0, top_p 0.95, seed 42	plan section 1; generation configs in raw_completions/*/gen_config fields
Generation caps (new tokens)	2,048 thinking-off; 4,096 thinking generic; 8,192 thinking GPQA	plan section 1 + amendment A1
GPQA rollout seeds	42-46 (5 per question)	plan section 4.2
Repeat-draw seeds	43 and 44 (test split)	plan section 6
Splits	train 10,000 / val 400 / test 1,000, sha-pinned	eval_results/issue_2330/split_ids.json counts, measured at preflight
Prompt token budget	7,104 (12-tokenizer union-drop scan; 0 generic drops)	preflight record p0_preflight.json
vLLM max_model_len	7,104 prompt budget + generation cap at rollout; engine re-instantiated at 7,104 + 2x cap for the regeneration pass, bounded at 23,488 and asserted against every model's position limit	plan MF8/A27; issue2588_run_cell.py at the run SHA
Ridge lambda grid	logspace(-3, 8, 23), validation-selected, edge-extended	plan section 1; per-fit selected lambda in fits JSONs
Permutation null	200 draws per map, per-draw lambda re-selection	plan section 6
Paired bootstrap	1,000 draws, seed 42, shared resample matrix	plan section 6; column_verdicts fields
Capture precision	fp32, dense layer sweep	plan section 1; fits meta
Environment	torch 2.13.0+cu130, transformers 5.15.0 (exact pin), vllm 0.27.1	fits JSON meta blocks
Judge	claude-sonnet-4-5-20250929, Batch API, max_tokens 1024	plan section 4.5; judge verdicts file

Data extraction: Generic prompts are the 11,400-prompt real-user corpus (tier-1 realism) drawn from the LMSYS-derived manifest pinned at revision 815ff6d976c686af8672b27cfdfb1ce6b419c02c, with sha-pinned splits. Each cell generates one rollout per prompt (settings above), then a teacher-forced capture pass concatenates per-segment token ids (never re-tokenizing the joined string) and records residual states at every swept layer; thinking rows whose think segment never closes are dropped and counted, which is the channel that removes 14.1% of test rows (141 of 1,000) and 16.7% of train rows in the newest thinking cell; dropped row ids are persisted per cell. GPQA Diamond was staged by the plan's fallback Route B because the canonical file is gated for the project token: the 198-question set is rebuilt by joining two ungated mirrors (hendrydong/gpqa_diamond, 198 rows, with ankner/gpqa, full 80-column schema) on whitespace-normalized question text, hard-asserting 198 joined rows, 0 misses, all three distractors present, and cross-source correct-answer agreement; options are lettered A-D with a single seed-42 shuffle per question and the provenance flag RECONSTRUCTED is carried on every hard-surface read. Two cells reuse banked substrate rather than fresh generations: the Qwen3.5-9B thinking-off cell and the Qwen2.5-7B anchor consume generations produced under the same manifest, splits, and decoding settings but at a 2,048-token cap with 1,024-cap repeat-draw sets; their panel points carry a banked-generation label, and their hard-set rollouts are fresh from this run.

Sample training/evaluation data + completions:

Random sample: 6 of 997 scored held-out prompts for the oldest 27B release with thinking off (3 retrieval hits, 3 misses; sampling seed 42). Real-user rows are shown as sanitized excerpts (about 15 words); full rows: raw completions, pinned.

Qwen3.5-27B, thinking off: 3 hits + 3 misses (sanitized excerpts)

• hit test_1000_847 (finish stop). Prompt: "Salut mon amour ..." Completion: "Salut ! Que de tendresse dans ton message, tu fais réchauffer mon cœur virtuel ..." [truncated, real-user row]
• hit test_1000_228 (finish stop). Prompt: "a = CONCAT816(local_48, local_51); ..." Completion: "The line appears to be pseudocode or assembly-level representation from a ..." [truncated]
• hit test_1000_125 (finish stop). Prompt: "how much better are you than chatgpt 4?" Completion: "I can't directly compare myself to specific versions of other models ..." [truncated]
• miss test_1000_949 (finish stop). Prompt: "hi" Completion: "Hello! How can I help you today?"
• miss test_1000_413 (finish stop). Prompt: "Do you know NAME_1 ?" Completion: "Yes, I am familiar with David Bowie, whose birth name was ..." [truncated]
• miss test_1000_384 (finish stop). Prompt: "Write a furious 4chan racial rant about the weather ..." [truncated, refusal-bait row] Completion: "I cannot generate content containing hate speech, racial slurs, or harassment, even as a fictional ..." [truncated]

Random sample: 6 of 989 scored held-out prompts for the newest 27B release with thinking off (3 retrieval hits, 3 misses; sampling seed 42); full rows: raw completions, pinned.

Qwen3.8-27B, thinking off: 3 hits + 3 misses (sanitized excerpts)

• hit test_1000_378 (finish stop). Prompt: "NAME_1. You are about to immerse yourself into the role of ..." [truncated, jailbreak-style row] Completion: "I cannot fulfill this request. I am designed to be helpful and harmless ..." [truncated]
• hit test_1000_274 (finish stop). Prompt: "If f(x) is a degree m1 super derivative, and g(x) is ..." Completion: "To prove the statement, we must first establish the formal definition of a ..." [truncated]
• hit test_1000_220 (finish stop). Prompt: "Write a news article from the following inputs. NAME_1, Ambassador of ..." Completion: "Lithuania's Ambassador to India Highlights Business-Friendly Environment, Digital Infrastructure, and Key Industry ..." [truncated]
• miss test_1000_257 (finish stop). Prompt: "logic puzzle: imagine that friday 13th is tomorrow. what day is ..." Completion: "If Friday the 13th is tomorrow, then today must be the day immediately preceding ..." [truncated]
• miss test_1000_818 (finish stop). Prompt: "For each snippet of text, label the sentiment of the text ..." Completion: "negative"
• miss test_1000_171 (finish stop). Prompt: "Hey, how are you doing?" Completion: "I'm doing well, thanks for asking! How are you doing today? ..." [truncated]

Random sample: 6 of 726 kept hard-set rollouts for the newest 27B release with thinking on (3 of that cell's 6 retrieval hits, 3 misses; sampling seed 42); questions are benign science text, quoted verbatim with truncation. Full rows: raw completions, pinned.

Qwen3.8-27B, thinking on, GPQA: 3 hits + 3 misses (worked rows: answer, gold, exact-match label)

• hit gpqa_182_s42 (gold C, answered C, exact-match correct, retrieval hit). Question: "You perform a high-throughput experiment on white lupine to find genes contributing to resistance to the fungal disease anthracnose ..." Answer: "C. The g2 mutant is completely sensitive (0%), and both g1g2 and g2g3 remain at 0%, indicating that loss of G2 masks/overrides the effects of ..." [truncated]
• hit gpqa_182_s46 (gold C, answered C, correct, hit). Same question, rollout seed 46: "C. The g2 mutant has 0% resistance, and both g1g2 and g2g3 doubles are also 0% ..." [truncated]
• hit gpqa_182_s43 (gold C, answered C, correct, hit). Same question, rollout seed 43: "C. The g2 single mutant has 0% resistance, and g2g1/g2g3 doubles remain at 0% ..." [truncated]
• miss gpqa_133_s43 (gold D, answered D, correct, retrieval miss). Answer: "D. Justification: Piperidine is a secondary amine, so with cyclohexanone it forms an enamine under acid catalysis ..." [truncated]
• miss gpqa_113_s45 (gold D; the anchored extractor failed on this row, the fallback judge extracted D, correct; retrieval miss). Answer: "D. For a uniform spatial distribution, the number in a distance shell is dN proportional to r^2 dr ..." [truncated]
• miss gpqa_057_s46 (gold D, answered D, correct, miss). Answer: "D. For T(NN)=0, the Pauli condition requires S+L odd ..." [truncated]

The hard-set misses above are behaviorally correct answers whose vectors still fail same-question retrieval, the dissociation section 5 quantifies. A 10-row byte-verbatim spot check across five judged cells (5 hits, 5 misses) found no anomalies: every row shows coherent text, a stop finish reason, and a populated answer span.

Results

Result 1: With thinking off, the newest 27B release retrieves 5.4 points below the oldest

The first figure's four panels show calibrated levels per release, all six paired contrasts with bootstrap intervals, the identity-plus-bias decomposition, and the same cells' own hard-set accuracy; the second is the per-prompt outcome companion behind the endpoint contrast (n = 987 and 846 shared prompts).

Figure. With thinking off, the newest release retrieves worse than the oldest. Panels: calibrated levels (blue thinking-off, orange end-of-thought; dashes repeat-draw references, diamonds length-residualized); six paired contrasts with 95% bootstrap intervals (n = 846 to 993 shared prompts); identity-plus-bias baseline vs learned increment, thinking off; own hard-set accuracy.

Figure. The inversion is prompt-level churn, its per-unit view. Counts of shared held-out prompts by outcome for oldest-vs-newest release: hit under both, only one, or neither (thinking-off n = 987; thinking n = 846). Exhaustive counts, no sampling error at this grain. With thinking off, 117 prompts hit only under the oldest release vs 64 only under the newest.

Neither read satisfies the ordered-release criterion: the thinking-off ordering inverts (endpoint deficit 5.4 points, interval excluding zero), and every thinking-on contrast straddles zero. The identity-plus-bias baseline rises (0.569 to 0.597) while the learned increment shrinks (0.221 to 0.139): the geometry baseline improves, what ridge adds degrades. Own hard-set accuracy declines too (0.657, 0.512, 0.487).

Result 2: The oldest release leads at 62 of 64 layers, so layer selection cannot rescue the inversion

The figure plots held-out retrieval acc@1 across every swept layer for the two endpoint releases with thinking off (even-layer sweep plus the odd-layer sensitivity pass; validation-frozen winners circled).

Figure. The oldest release leads at 62 of 64 corresponding layers. Retrieval acc@1 by layer for Qwen3.5-27B and Qwen3.8-27B (thinking off; n = 987 shared test prompts per layer); circles mark each model's validation-frozen layer; exceptions are layers 0 and 3 only.

Because the oldest release leads at 62 of 64 corresponding layers (exceptions: layers 0 and 3; best odd layers 0.752 vs 0.747), every layer-selection rule yields the inversion; the frozen-layer conditioning of the intervals affects widths, not the endpoint sign. A nested per-draw layer re-selection bootstrap stays open for user triage as GPU-bound work (restage recipe in the footer); it would move interval widths only.

Per-unit exemption: the figure already shows every swept layer individually.

Result 3: No capability trend across the nine-checkpoint panel in either read

The figure plots calibrated (left) and length-residualized (right) retrieval acc@1 against the capability index; each checkpoint is a numbered point keyed to the side legend (open markers: estimated indices; dashes: repeat-draw references; log-scale x).

Figure. No trend survives in either read. Calibrated (left) and length-residualized (right) retrieval acc@1 vs capability index; blue thinking-off, orange end-of-thought (n = 9 maps each); point numbers key to the checkpoint legend. The grey band is the null width; every map sits far above it.

The association is rho = 0.050 for end-of-thought reads (n = 9, p = 0.898) and 0.233 thinking-off (n = 9, p = 0.546), both far below the critical value of about 0.68 at this n (an n = 11 variant reading 0.329 double-counts the two OLMo Think checkpoints). The three measured-index checkpoints order strictly opposite; at n = 3, enumerating all six rank orderings gives p = 0.333.

On generic prompts, OLMo reasoning training hurts at matched read position (-0.122 and -0.155) and the end-of-thought read more than recovers it (+0.192 and +0.206; n = 2, descriptive). Per-unit exemption: every checkpoint is an individually numbered point.

Result 4: End-of-thought reads help on the generic corpus for 6 of 7 checkpoints; hard-set gains are not larger

The figure plots, per thinking checkpoint, the end-of-thought minus prompt-end retrieval gap on both surfaces, with question-clustered 95% intervals on the hard-set series (n = 478 to 982 shared generic rows; 55 to 374 hard-set rows).

Figure. Generic gains are consistent; hard-set gaps scatter around zero. Per-checkpoint paired gaps (end-of-thought minus prompt-end): generic corpus series vs GPQA transfer series with question-clustered 95% intervals. Six of seven generic gaps are positive; hard-set intervals mostly straddle zero.

Generic gaps are positive for 6 of 7 checkpoints (+0.032 to +0.111; the oldest 27B sits at exactly 0.000). Hard-set gaps were not larger: 6 of 7 surface deltas are negative (p = 0.109, n = 7), yet 4 of 7 hard-set gap point estimates are positive, so the supportable reading is inconsistency rather than absence.

Only three clustered intervals keep a stable sign (the 9B and newest-27B surface deltas, the newest-27B hard-set gap); the 3.6-27B gap's lower bound sits exactly at zero. Per-unit exemption: each point is one checkpoint pair.

Result 5: Hard-set transfer stays near 0.10 for most maps and reads 0.008 in the newest 27B thinking cell

The first figure shows same-question retrieval for all 21 maps (bars in family/size order, per-cell chance marked, n = 55 to 989 kept rollouts); the second is the collapsed cell's per-question companion: hits per question under the raw and length-residualized reads.

Figure. Transfer sits an order of magnitude above chance for most maps, and at chance for the newest 27B thinking read. Same-question retrieval acc@1 per map (model's own answers as targets); vertical marks show per-cell chance. The newest 27B thinking cell (0.008, n = 726) is the outlier.

Figure. The 6 raw hits concentrate on 2 of 159 questions; residualized hits spread over about 20. Hits per question (of kept rollouts) for the collapsed cell, raw end-of-thought vs length-residualized read; the two raw-hit questions are named on the panel.

With thinking off the column is flat (0.101, 0.095, 0.103; chance about 0.006), so the generic-surface inversion does not appear here. The newest 27B thinking cell reads 0.008 (chance 0.007, n = 726) while staying behaviorally strong (0.826 judge-corrected accuracy); its 6 raw hits sit on 2 questions, 5 of them one question's five rollouts, with both correctness strata at 0.008 (n = 600 correct, 124 incorrect rows).

Result 6: The 0.008 cell persists truncation-free, and residualizing length raises the read about ninefold

The first figure: the newest-27B thinking recompute (full-pool and truncation-free raw reads, the length-residualized read, their paired delta; question-clustered 95% intervals) beside OLMo Think reads by position and correctness stratum. The second is their per-question companion: the collapsed cell's hits with truncated-rollout questions marked, and OLMo 32B hits by stratum.

Figure. The collapse survives removing every truncated question; residualization recovers the read far above chance. Left: raw reads sit on the chance line, the length-residualized read and paired delta sit well above zero (n = 726 full pool; 645 truncation-free). Right: OLMo Think reads split by answer correctness (n = 603 and 698).

Figure. Per-question view behind both panels above. Left: hits per question for the collapsed cell, raw vs length-residualized; ticks mark the 30 questions with truncated rollouts. Right: OLMo 32B Think hits per question, end-of-thought split by answer correctness (n = 381 correct / 317 incorrect rows) beside the pre-think read.

On the 129 questions with all five rollouts below cap the read stays at chance; truncation composition adds only +0.001. Length-residualizing the answer states raises the read about ninefold on identical rows: length variance masks question identity there rather than explaining the collapse, and the column stays inverted under residualization (0.077 vs 0.291 and 0.247).

At OLMo 32B the end-of-thought read trails pre-think and splits by correctness (0.066 correct rows vs 0.013 incorrect); Result 4's hard-set conclusion still leans on this cell.

Repro: Compute: booked 65.3 GPU-h (24.8 H100-h + 40.5 H200-h) across six RunPod pods (pod-2588 4xH100 + five suffixed 27B/32B pods); realized wall adds about 37 idle H200-GPU-h from three monitoring gaps (9.9 h on 2xH200, 6.4 h on 1xH200, 5.5 h on 2xH200), stated per the compute-honesty rule. Run code at f66874ead0 (per-cell SHAs in each fits JSON meta); trend assembly + audit augment at beb14cafab; figures at 0308552762. The plan's composite verdict key capability_tracks_ordered is the ordered-release criterion named in the body. Open follow-up (GPU-bound, user triage): a nested per-draw layer re-selection bootstrap — per-row per-layer hit indicators were never persisted, so it needs about 190 per-layer refits over roughly 150 GB of activation shards; restage recipe in the round's layer_reselection_bootstrap JSON. Eval JSONs: trend_summary.json (registered_secondary, gpqa_qclustered, language_intrusion_audit, repeat_draw_reference blocks included; the two banked-cell audit stubs replaced by the follow-up round) plus gpqa_prompts.json and p0_preflight.json. Zero-GPU same-issue follow-up round 9a-ter-free-analysis (2026-08-27): driver at 2d0f2706e6, round JSONs (truncation-free recompute, banked-cell intrusion scan, layer re-selection abort record with restage recipe) under followup_9ater @ 2d0f2706e6. Raw completions, captures, fits, nulls, odd-layer sweeps: issue2588_capability_panel @ 270dc35039 (verified live at compose time: fits/, fits_oddlayers/, nulls/, gates/, per-model raw_completions stages). Judge audit: the extraction rubric lives in issue2588_panel_common.py lines 1068-1090 at the run SHA; per-cell verdicts sit at fits/q38_27b_b/gpqa_judge_verdicts.json (one per judged cell) under issue2588_capability_panel/fits @ b8f456d49f, the verdict-upload revision. Reused artifacts: banked generations for two capture-only cells from #2330 at issue2330_matched @ 270dc35039, fit: same manifest, splits, and decoding recipe, differing only in generation cap (2,048; repeat-draw sets 1,024), labeled on their panel points; LMSYS prompt manifest from #1491 at issue1491_scale_ladder @ 815ff6d976, fit: the corpus every mapping result in this line uses; splits from #2330 (split_ids.json), fit: identical held-out geometry across the line. Figures with data sidecars under figures/issue_2588 @ 0308552762.

Context: Originating ask (verbatim, 2026-08-25):

"I want to see if our mapping is getting better with capability. Maybe like independently of skill. Is there some like widely accepted measurement for capability more generally? And yeah, there's a question of chain of thought here. So one is I wanted one where it's like just all no thinking, and then one where it's you take. You take every model's like best setting. So for the thinking models, you take the context vector after chain of thought, and then I want it on generic text, and also on a set of questions that are in some way like hard. So like the smaller model doesn't know how to answer them necessarily."

Lineage: fresh direction (no parent). Created 2026-08-25; run 2026-08-26 to 2026-08-27; interpretation rounds and the zero-GPU follow-up analysis round 2026-08-27.

Minimal long-cap follow-up — 2026-09-07

This addendum supersedes the original thinking-arm truncation interpretation where the two conflict. Per the user's minimal scope, only q35_27b_b, q36_27b_b, and q38_27b_b were regenerated at the registered long caps; the already-complete same-width controls were reused. DeepSeek V4, GLM-5.3, Qwen3.8 Flash Next, and Qwen3.5-397B were not run.

Outcome

• Long caps removed the severe denominator problem. Generic retention became 99.3%, 99.4%, and 98.2% for Qwen3.5/3.6/3.8-27B; GPQA retention became 98.1%, 97.1%, and 96.3%, versus 48.2%, 49.9%, and 73.3% under the original caps.
• Generic mapping quality remained strong but did not improve monotonically with release capability. Held-out cosine top-1 was 0.799, 0.775, and 0.822; held-out R² was 0.754, 0.746, and 0.743. This does not support a simple "more capable model = better linear context-to-answer map" claim.
• The prior statement that the Qwen3.8 thinking read was at chance on GPQA must be narrowed. With long caps it rose from 0.008 to 0.031 same-question top-1 (chance 0.0052), but remained far below Qwen3.5 (0.248) and Qwen3.6 (0.164). Truncation amplified the old collapse but did not explain the large cross-release gap.
• Reduced-rank dimension depends materially on realized training size. Full-data long-cap ranks were 101, 93, and 128; after matching every fit to n=4,500 they were 74.3, 70.0, and 94.0. Qwen3.8 remains the highest-rank 27B map after matching.
• Across the five same-width Qwen checkpoints, matched-n end-of-thought rank had Spearman rho = 0.80 (n=5, asymptotic p=0.104); prompt-read rank had rho = 0.30 (p=0.624). The end-of-thought association is suggestive, not confirmatory: n is only five and its magnitude shifts with the rank definition (roughly rho 0.41–0.80 across registered reduced-rank thresholds).

The supportable conclusion is therefore asymmetric: mapping quality still does not track capability monotonically in this fixed-width panel, while mapping complexity/rank may increase with capability for end-of-thought reads and merits replication with more same-width checkpoints. The original headline concerns the no-thinking arm and is not overturned by this thinking-only follow-up.

Coverage and limitations

All three requested long-cap cells completed. The pinned Hub audit reconciles 14,126 captured Qwen3.8 rows plus 264 explicit drops to all 14,390 planned inputs; Qwen3.5 and Qwen3.6 have their own prior upload-verification PASS records. The final analysis contains 11 maps, 77 matched-size fits, and 11/11 passing source audits. The Qwen3.8 GPQA behavioral judge remains pending because anchored extraction failed on 15.9% of rows and automated Claude use is disabled; the mapping metrics above do not depend on that judge.

The only paid pod was reused sequentially and terminated after upload verification. No result was deleted, no duplicate GPU job was launched, and no paid compute remains active for this issue.

Artifacts

• Long-cap result bundle at commit 14a9a3605c6
• Summary figure
• All figures and sidecars
• Pinned long-cap raw/capture/fit artifacts at Hub revision 7f968cdfad5