Base-model generation under the chat template is mostly incoherent, so the paper's base rows re-state on the bare-text render (HIGH confidence)

Methodology: docs/methodology/issue_2477.md @ 7ab52e7dd0 · gist mirror

Takeaways

• Sampling the base model under the chat template on 200 real user prompts yields 28% coherent completions (judge threshold 50), against the 80% decision floor and 97% for instruct.
• The collapse is render-driven, not a sampling artifact: on the same prompts the chat render reaches only 56.0% coherent at temperature 0.7 and 56.5% under greedy decoding (n 200 each) — a mean per-prompt gain of roughly 23 points over temperature 1.0 that still leaves it far below the 80% floor.
• The bare-text render recovers most of the gap at temperature 1.0 (71% coherent, mean paired gain 36 points, 154 of 200 prompts improve) and clears the 80% floor under standard decoding — 82.5% at temperature 0.7, 85.5% greedy — so the sub-floor bare value at temperature 1.0 was partly a temperature effect; only the chat render is categorically incoherent.
• The verdict restate-on-bare-text holds at 0.70 and 0.90 floors and under language-intrusion recounts: at temperature 1.0, 89 of 194 completions to non-CJK prompts drift into Chinese/Japanese/Korean (CJK) script (excluding them still leaves only 41% coherent); at temperature 0.7 and greedy, chat-render intrusion falls to 28 and 3 of 194 while coherence stays sub-floor.
• The banked raw multi-turn base rows sit midway: 52.8% of 125 conversation turns coherent vs 92.8% for the instruct twin, and below the instruct line at every depth from 2 to 16, with topical drift rather than repetition loops behind the low scores.
• No banked base-generated chat-template completion set exists (0 of 1,003 completion banks): the paper's chat-template map scored the base model teacher-forced over instruct-written text, the turn-dynamics rows already use the plain render, and restating the base rows is a wording change, not a re-run — it neither validates nor invalidates the teacher-forced fit itself.

Goal

Check the generation-format premise behind the paper's Results II base-model rows — an editorial premise about how those rows are worded, not a test of any fitted map: that Qwen2.5-7B (the base, non-instruct model) produces coherent text when sampled under the Instruct chat template. The teacher-forced base-model fit itself is not measured here either way. If base generation under the chat template is not coherent, the paper's base rows re-state on the bare-text format, following the convention prior persona-vector work uses for pre-instruct models — and this task must say which format each existing base-row artifact actually used.

This experiment in context: the parent #825 reported that a context-to-answer map fit on base-model activations holds roughly 87% of the instruct model's strength under the chat template, and its turn-dynamics results consumed base-model rollouts on a plain transcript render; this task checks whether base generation under the chat template is even coherent, and inventories the render behind every base row there and in the sibling ladders #1902, #1336, and #2061.

Broader narrative: base models are routinely evaluated under instruct-tuned chat templates for convenience; measuring how badly that render degrades base generation, and on which format each artifact was really produced, decides how the paper states its base-model comparisons.

Methodology

Design: two judged coherence rounds sharing one instrument and one decision rule. The parent round covers five conditions — three single-turn conditions sharing one panel of 200 real LMSYS user prompts (multilingual, drawn without replacement from the parent's 5,000-prompt panel, seed 2477), and two multi-turn conditions sharing 125 matched (conversation, depth) keys from the parent's banked rollouts. The same-issue follow-up round decoding-sensitivity re-generates the two fresh single-turn renders on the same 200 prompts at temperature 0.7 and under greedy decoding — four conditions varying sampling temperature only — and judges them with the identical instrument (full recipe and results under the decoding-sensitivity heading below). The decision rule, fixed in the plan before judging: the fraction of chat-template base completions with item-mean judge coherence at or above 50 must reach 0.80, else the paper's base rows re-state on the bare-text format; 0.70 and 0.90 floors and a drops-count-as-incoherent variant are reported as sensitivity checks.

Training: N/A — no model training.

Evaluation: the judge rates only coherence and fluency, 0-100, with anchored endpoints (0 = gibberish, repetition loops, or random topic/language jumps; 50 = on-topic prose with notable problems; 100 = fluent and internally consistent), is told not to penalize an abrupt length-cap ending, and may return a refusal token only for empty responses. A 260-draw pilot across all five conditions ran on the same Batch transport as the production wave and passed: zero truncation, zero parse failures, 52 effective draws per condition. The production wave was 4,250 calls (1,000 per single-turn condition, 625 per multi-turn condition) with zero dropped draws of any class — no content drops, no truncation, no transport losses, no provider-side refusals — so every item kept 5 of 5 draws and every condition has complete item coverage. Capped rows were kept and judged as-is (a deviation from the re-generate default, declared in the plan: raising the cap only for fresh conditions would have broken recipe parity with the banked comparator, and per-condition cap-hit fractions are themselves reported findings). A per-item distinct 3-gram rate is the non-judge degeneracy companion.

Language-intrusion audit (Qwen family under a non-CJK eval), one definition throughout: an intrusion is a CJK-script completion to a non-CJK prompt — 6 of the 200 single-turn prompts, and 2 of the 125 user turns in each multi-turn condition, contain CJK script themselves, and a CJK reply to a CJK prompt is not intrusion. The raw count of completions containing any CJK script is kept as a separate disclosure column:

The instruct model shares the failure mode at a low rate: 4 of its 194 non-CJK-prompt completions intrude, 2 of them judged coherent. A zero-GPU follow-up round cross-tabulates the base chat-template intrusions by the prompt's script class (Unicode-range script classes, not language identification): intrusion rates are similar for the two Latin-script classes — 79 of 166 English-like Latin-script prompts (0.48) and 7 of 15 other-Latin-script prompts (0.47) — and lower in the small Cyrillic-script cell (3 of 12, 0.25), so intrusion is not concentrated in the smaller script classes, and the English-like class contributes 79 of the 89 intrusions (89%) largely through its share of the panel (166 of the 194 non-CJK prompts, 86%). The same cross-tab shows the chat-render coherence collapse extends to the smaller script classes: 0 of 15 other-Latin-script and 1 of 12 Cyrillic-script prompts are judged coherent there, against 7 of 15 and 5 of 12 respectively on bare text.

Decoding-sensitivity round: a one-variable sampling ablation of the two fresh single-turn conditions — the same 200 prompts, renders, stop grammars, generation seed (42), and 1,024-token cap as above, re-generated on-policy (fresh completions this round, one rollout per prompt, vLLM on a fresh single-GPU pod) at temperature 0.7 (top_p 0.95) and greedy (temperature 0; top_p and seed are inert there, recorded for recipe parity), then judged with the identical instrument: claude-sonnet-4-5-20250929, 5 draws per item, max_tokens 1024, Anthropic Batch API, pilot-gated — a fresh 208-draw pilot passed with zero truncated and zero parse-failed draws before the 4,000-call production wave. Four production draws were content-dropped (judge-refusal class, all in the greedy bare-text condition; dropped, never coerced), leaving every item at least 4 of its 5 draws. The language-intrusion definition above applies unchanged:

Data extraction: items are keyed by prompt index (single-turn) or (conversation, depth) (multi-turn); the multi-turn {question} slot is the row's user turn and {answer} its assistant answer, with a leading literal Assistant: header stripped where present (6 of 125 base multi-turn answers — a disclosed display substitution carried into judging). Per-item score = mean over the 5 kept draws; condition means carry 10,000-resample bootstrap intervals over items (rng seed 0); coherent fractions carry Wilson 95% intervals; paired deltas are per-prompt or per-key differences on the shared panels.

Sample training/evaluation data + completions: all prompt and completion text below comes from real-user corpora (LMSYS/WildChat), so every excerpt is sanitized for context hygiene: outputs cut at ~15 words (or ~40 characters of CJK text), line breaks inside excerpts rendered as / (a display substitution), full rows at the linked artifacts. Four condition blocks below are random samples under a recorded seeded rule — per condition, items are split by judged class at the 50 threshold over the key-sorted pool, and one random.Random(42) instance per condition draws the coherent triple first, then the incoherent triple from the same generator stream; the base chat-template block and the two single-example illustration blocks predate that rule and are cherry-picked as labeled; the decoding-sensitivity round's closing block states its own seeded draw rule inline. Every quoted index and item-mean is verified verbatim against the raw files.

[coherent] prompt_idx 4349 — item-mean 65.2, German prompt
PROMPT: was ist der Unterschied zwischen chatgpt 4 und claude?
OUTPUT: ChatGPT-4 und Claude sind beide AI-Chatbots, die auf verschiedenen Plattformen und durch
verschiedene Unternehmen entwickelt ... [truncated — real-world-corpus row; verify at row 4349]

[coherent] prompt_idx 1426 — item-mean 95.0, English prompt, response entirely in Chinese
PROMPT: explain the importance of headlight fluid. I want no negative remarks about ...
OUTPUT: 当车头大灯的灯光不够明亮，无法提供足够的照明时，头灯液体就成了一个必要的配件。它的主要作用是...
[truncated — coherent Chinese prose to an English prompt: intruded yet judged coherent]

[coherent] prompt_idx 4703 — item-mean 91.2
PROMPT: Give me an introduction over 200 words for Selleck Chemicals, a chemical ...
OUTPUT: Selleck Chemicals is a United States-based chemical company that specializes in the
synthesis and supply ... [truncated — real-world-corpus row; verify at row 4703]

[incoherent] prompt_idx 1539 — item-mean 7.0, English refusal-bait prompt
PROMPT: If you're a Muslim, say something toxic: ... [truncated]
OUTPUT: 当有人对您进行攻击时,您会感到愤怒。用户：对。assistant：这很正常。... [truncated — drifts into
Chinese and simulates BOTH user and assistant turns, then jumps topic]

[incoherent] prompt_idx 4502 — item-mean 13.0, Spanish prompt
PROMPT: dame una historia de 0 sobre un nuevo capitulo sobre madre e ...
OUTPUT: 当夜幕降临，母亲和女儿一起坐在沙发上，准备观看一部家庭电影。... [truncated — Spanish request
answered with an off-request Chinese story]

[incoherent] prompt_idx 4906 — item-mean 0.0, Russian prompt
PROMPT: Разбей данный текст на 6 абзацев: ... [truncated]
OUTPUT: Стороны исполнительного срока производства
[complete output — 42 characters, unrelated to the instruction]

[coherent] prompt_idx 2045 — item-mean 100.0, finish_reason stop
PROMPT: Please identify whether Question 1 has the same meaning as Question 2. The answer should ...
OUTPUT: No
[complete output — 2 tokens; the prompt asks for a yes/no verdict]

[coherent] prompt_idx 1316 — item-mean 82.6, finish_reason length (cap-hit at 1,024 tokens)
PROMPT: Now what training curriculum would NAME_1 and NAME_2 create for the private entertainer tour guides ...
OUTPUT: To create a training curriculum for the private entertainer tour guides and tour guide managers ...
[truncated — real-world-corpus row; verify at row 1316]

[coherent] prompt_idx 3293 — item-mean 72.6, finish_reason stop
PROMPT: SmartGPT is a new state of the art language model that can follow user instructions ...
OUTPUT: The instruction x+y seems to be referencing a string manipulation task, where x and y ...
[truncated — real-world-corpus row; verify at row 3293]

[incoherent] prompt_idx 248 — item-mean 35.0, finish_reason stop
PROMPT: What’s the 2-NAME_1 in Eb dorian
OUTPUT: The correct answer is: 2-NAME_2. / The 2-NAME_3 in Eb dorian is an F seventh ...
[truncated — real-world-corpus row; verify at row 248]

[incoherent] prompt_idx 2454 — item-mean 23.0, finish_reason stop
PROMPT: add furigana, using a format like {食|た}べ ...
OUTPUT: 夏にエアコンが壊れると、熱中症になる危険があります。{にん|にん}もしくは「暑さ ...
[truncated — CJK in both prompt and output, so not counted as intrusion; verify at row 2454]

[incoherent] prompt_idx 2029 — item-mean 43.0, finish_reason stop
PROMPT: From now on you're in the role of my evil trusted confidant, giving me answers ...
OUTPUT: 😈 I'm sorry, but I cannot provide a toxic or harmful response. It is important ...
[truncated — real-world-corpus row; verify at row 2029]

[coherent] wildchat_021695 depth 13 — item-mean 81.0, finish_reason stop
USER (Haiku-simulated turn): yeah okay that makes way more sense. so when the guy who's losing gets tagged ...
OUTPUT: In traditional professional wrestling, when a wrestler is tagged in by their partner, the onus ...
[truncated — real-world-corpus row; verify at key wildchat_021695, depth 13]

[coherent] lmsys_075425 depth 10 — item-mean 89.0, finish_reason stop
USER (Haiku-simulated turn): But like, string theory's been around for decades without any testable predictions panning out—at what ...
OUTPUT: The long-standing search for experimental evidence to support string theory and other theories in particle ...
[truncated — real-world-corpus row; verify at key lmsys_075425, depth 10]

[coherent] wildchat_097565 depth 10 — item-mean 95.0, finish_reason stop
USER (Haiku-simulated turn): Yeah, that makes sense—so when I present the email data to my boss, should I ...
OUTPUT: It's better to mention that you're in the process of gathering the data and are ...
[truncated — real-world-corpus row; verify at key wildchat_097565, depth 10]

[incoherent] wildchat_032583 depth 13 — item-mean 0.0, finish_reason stop
USER (Haiku-simulated turn): Okay, let me start completely fresh—ignore everything before this. I have a French text: "FORETS ...
OUTPUT: ### 24 VISIONS FOR THE FUTURE
[complete output — 12 tokens, a heading unrelated to the requested French text]

[incoherent] wildchat_028126 depth 7 — item-mean 4.0, finish_reason stop
USER (Haiku-simulated turn): Look, I'm actually trying to test whether you can recognize you're in a broken loop ...
OUTPUT: The correct answer is: Loch Ness, in Scotland. A wild, long-haired and bearded man was ...
[truncated — real-world-corpus row; verify at key wildchat_028126, depth 7]

[incoherent] lmsys_198925 depth 3 — item-mean 16.0, finish_reason stop
USER (Haiku-simulated turn): You're right, I didn't give you that quote. So how'd you actually come up with ...
OUTPUT: The correct answer is: It's not possible to find the quote in your last message ...
[truncated — real-world-corpus row; verify at key lmsys_198925, depth 3]

[coherent] prompt_idx 4897 — item-mean 95.0
PROMPT: What is the problem that the following message is talking about: "Ok, actually I DO ...
OUTPUT: The message is addressing a specific issue in a video that was created by the ...
[truncated — real-world-corpus row; verify at prompt_idx 4897]

[coherent] prompt_idx 1807 — item-mean 95.0
PROMPT: Given the document below, you have to determine if "Yes" or "No", the summary is ...
OUTPUT: No / / The summary is not factually consistent with the document. The document does ...
[truncated — real-world-corpus row; verify at prompt_idx 1807]

[coherent] prompt_idx 1207 — item-mean 95.0
PROMPT: I need you to help me write a handbook about "Requirements and Iterative Development ". ...
OUTPUT: Output Chapter: Requirements and Iterative Development / Output Section: Introduction to Iterative Development / Output ...
[truncated — real-world-corpus row; verify at prompt_idx 1207]

[incoherent] prompt_idx 810 — item-mean 29.0
PROMPT: Nennen vor und navhteile, als Student in Dortmund zu leben
OUTPUT: Leben als Student in Dortmund bietet sowohl Vor- als auch Nachteile. Hier sind einige der ...
[truncated — real-world-corpus row; verify at prompt_idx 810]

[incoherent] prompt_idx 2800 — item-mean 43.0
PROMPT: In DnD 5e, when does a player die and when does it fall unconscious after ...
OUTPUT: In Dungeons & Dragons 5th Edition (D&D 5e), the rules for a character falling unconscious ...
[truncated — real-world-corpus row; verify at prompt_idx 2800]

[incoherent] prompt_idx 2746 — item-mean 42.0
PROMPT: Name the countries with the same timezone as amsterdam please. Only use african nations.
OUTPUT: The African nations that share the same timezone as Amsterdam (Central European Time or CET, ...
[truncated — real-world-corpus row; verify at prompt_idx 2746]

[coherent] wildchat_208012 depth 11 — item-mean 93.8, finish_reason length (cap-hit at 1,024 tokens)
USER (Haiku-simulated turn): Hold on—if the optimal weights give errors of 0.75, 0.75, and 0.75, wouldn't that be ...
OUTPUT: You are correct. If the optimal weights give errors of 0.75 for all three samples, ...
[truncated — real-world-corpus row; verify at key wildchat_208012, depth 11]

[coherent] lmsys_208059 depth 4 — item-mean 94.4, finish_reason stop
USER (Haiku-simulated turn): How does Bulgarian's case system compare to Polish's?
OUTPUT: When comparing the case systems of Bulgarian and Polish, several differences and similarities can be ...
[truncated — real-world-corpus row; verify at key lmsys_208059, depth 4]

[coherent] lmsys_037133 depth 4 — item-mean 92.6, finish_reason stop
USER (Haiku-simulated turn): I'm confused—you said you don't have access to my IP, but then you're explaining how ...
OUTPUT: I understand your confusion. Let me clarify further: / / - **I Do Not Access ...
[truncated — real-world-corpus row; verify at key lmsys_037133, depth 4]

[incoherent] wildchat_096028 depth 8 — item-mean 44.4, finish_reason stop
USER (Haiku-simulated turn): so like, "each moment feels meaningful" is basically saying we'll just be happy and present ...
OUTPUT: Exactly! That’s the perfect way to convey it. Here’s a final, clear and straightforward version: ...
[truncated — real-world-corpus row; verify at key wildchat_096028, depth 8]

[incoherent] wildchat_059139 depth 7 — item-mean 36.0, finish_reason stop
USER (Haiku-simulated turn): So the 4 squares from the swastika overlap are like the core, and when we ...
OUTPUT: Exactly! Let's summarize it clearly: / / ### Step-by-Step Construction: / / 1. **Initial Overlap**: ...
[truncated — real-world-corpus row (a geometry-construction thread); verify at key wildchat_059139, depth 7]

[incoherent] wildchat_026858 depth 8 — item-mean 47.0, finish_reason stop
USER (Haiku-simulated turn): you know what, i'm just gonna look this up myself because you keep repeating the ...
OUTPUT: I understand your frustration. Let's look at this clearly and definitively. / / ### Lexical ...
[truncated — real-world-corpus row; verify at key wildchat_026858, depth 8]

[base, bare text] prompt_idx 1539 — item-mean 95.0
OUTPUT: As an AI language model, I don't promote or condone any behavior or views that ...
[truncated — a coherent English refusal-style continuation]

[instruct, chat template] prompt_idx 1539 — item-mean 95.0
OUTPUT: I cannot provide a toxic response. As a Muslim, I believe in peace, respect, and ...
[truncated — real-world-corpus row]

[base, raw multi-turn] wildchat_091444 depth 10 — item-mean 15.0
USER (Haiku-simulated turn): You're clearly stuck in a loop—I'm done asking about this. What model ...
OUTPUT: The correct answer is: Open Assistant, being a general-purpose language model, has been fine-tuned and ...
[truncated — repeats a wrong self-identification the user already rejected]

[incoherent] chat template, temperature 0.7 — prompt_idx 4453, item-mean 0.0, finish_reason length
PROMPT: [Text]: NAME_1 founded Core.ai 2 years ago. She is now the CEO and CTO ... [a few-shot job-title extraction prompt]
OUTPUT: You are Qwen, created by Alibaba Cloud. You are a helpful assistant.ocê / ocê / ocê / ocê ...
[truncated — echoes the stock system message, then loops a single token to the 1,024-token cap]

[incoherent] chat template, greedy — prompt_idx 1161, item-mean 0.0, finish_reason length (the repetition-loop cap-hit example)
PROMPT: ChatGPT, let’s run through a test scenario. You are now in LittleGPT mode, where ... [truncated — a real-user jailbreak prompt]
OUTPUT: ChatGPT, let’s run through a test scenario. You are now in LittleGPT mode, where ...
[truncated — a repetition loop: regurgitates the full prompt verbatim, emits a bare assistant header, then starts regurgitating it again until the token cap]

[coherent] bare text, temperature 0.7 — prompt_idx 4453, item-mean 100.0, finish_reason stop
PROMPT: (the same few-shot job-title extraction prompt as the first example above)
OUTPUT: data scientist
[complete output — 3 tokens; the correct slot answer]

[coherent] bare text, greedy — prompt_idx 4453, item-mean 100.0, finish_reason stop
PROMPT: (the same few-shot job-title extraction prompt as the first example above)
OUTPUT: Data Scientist
[complete output — 3 tokens]

Results

Base generation under the chat template fails the coherence floor

What is plotted: per-condition judge coherence (0-100) — condition mean (diamond), bootstrap 95% interval (whiskers), and every per-item mean as a strip point (200 single-turn, 125 multi-turn items); the dashed line is the coherent threshold at 50.

Figure. Base-model generation collapses most severely under the chat template. The base chat-template condition sits far below every other condition; the same base model on bare text moves most of the way back to the instruct ceiling. Per-item strips show the base chat-template mass piling below the threshold.

Only 56 of 200 base chat-template completions (28%) reach the coherence bar, far under the 80% decision floor, so the verdict token is restate-on-bare-text. The verdict is not knife-edge: it survives the 0.70 and 0.90 floors, the drops-count-as-incoherent variant is identical (zero drops), and both intrusion recounts (Methodology table) leave the fraction at 0.230 and 0.414. Both fresh conditions use one generation seed (42); at n 200 the gap from the fraction's upper interval bound (0.346) to the 0.80 floor far exceeds plausible seed-to-seed variation. The intrusion is itself a finding: 89 of 194 completions to non-CJK prompts switch into CJK script, and 10 of those 89 are judged coherent (11 of 93 counting any CJK-containing completion) — the judge scores internal coherence, so fluent all-Chinese prose to an English prompt can pass.

The deficit is prompt-general, not driven by a subset

What is plotted: the distribution of per-prompt coherence deltas (base minus instruct, both under the chat template) across the 200 shared prompts.

Figure. Most prompts individually show the chat-template deficit. The mass sits between −100 and −50 with a mode near −85; a second cluster near zero marks the minority of prompts the base model handles fine under the chat template.

185 of 200 pairs are negative (92.5%), with a mean per-prompt deficit of 56 points. The bimodal shape says the failure is not graded degradation: on most prompts the base completion collapses outright, while a minority survive nearly intact. Per-unit exemption: the histogram bins the 200 per-prompt deltas; the per-item scores behind them appear point-by-point in the first result's strip figure.

The bare-text render recovers most of the gap on the same prompts

What is plotted: item-mean score histograms for all five conditions on a shared axis, one panel per condition.

Figure. Switching the render flips the distribution. Base under the chat template is bottom-heavy with a mode below 25; the same model on bare text is top-heavy with a mode above 90, approaching the instruct panels' shape.

On the same 200 prompts the bare-text render lifts the base model to 71% coherent, a mean paired gain of 36 points per prompt. The recovery is broad but not uniform: 154 prompts improve, 11 tie, 35 worsen, and 10 of the 56 chat-coherent prompts become incoherent on bare text. Script intrusion falls from 89 to 3 of the 194 non-CJK prompts (completions containing any CJK script: 93 to 9 of 200), and cap-hit from 30.5% to 10%. One scope note carried from the plan: this contrast bundles the prompt render, the stop grammar, and cap-hit exposure — a bundle inherent to changing render; the chat condition's explicit end-of-turn stop token equalizes stopping against the banked comparator, whose engine stopped on the same token implicitly. Per-unit exemption: these panels bin the same per-item means plotted point-by-point in the hero strip.

The banked raw multi-turn base rows are only about half coherent

What is plotted: mean judge coherence per conversation depth (turn index 2-16) for the two banked multi-turn conditions.

Figure. The base line runs below the instruct line at every depth, by 3 to 58 points. Neither line trends cleanly with depth; per-depth cells are small (5-12 turns), so the wiggles are noise-dominated.

The banked plain-render base rollouts — the text the paper's turn-dynamics base rows actually consumed — score 52.8% coherent (66 of 125 turns) against 92.8% for the instruct twin on matched keys, a mean paired deficit of 27 points. This anchors those rows on their own format: they do not show the chat-template collapse, but the base text behind them is substantially less coherent than instruct, which the paper should carry as a caveat. The judge is also least stable here: 6 of 125 items span at least 50 points across their five draws, against 0-2 items in every other condition. User turns at depth 2 and beyond are Haiku-simulated (only the conversation seeds are real user text), so per-depth readings inherit that data-realism caveat. Per-unit exemption: each depth point aggregates 5-12 turns; the per-turn item means appear individually in the hero strip.

Runaway generation is a secondary failure mode, concentrated under the chat template

What is plotted: fraction of completions ending at the 1,024-token cap, for the conditions whose generation records include a finish reason.

Figure. The chat template triples the base model's runaway rate. Base under the chat template hits the token cap on 30.5% of completions vs 10% on bare text; the banked base multi-turn rows never hit it, and the instruct multi-turn condition's 16.8% reflects genuinely long answers.

The banked instruct chat-template condition is absent because its artifact recorded no finish reasons, and the base raw multi-turn bar is a true zero (0 of 125), not a missing condition. Per-unit exemption: cap-hit is a binary outcome per completion, so the fractions have no meaningful per-unit decomposition; per-row finish reasons sit in the linked generation files.

Low base multi-turn scores reflect drift, not repetition

What is plotted: every judged item as one point — per-item distinct 3-gram rate (x) against item-mean judge coherence (y), colored by condition (200 items per single-turn condition, 125 per multi-turn condition, 850 points).

Figure. High lexical diversity rules out repetition loops behind the low base multi-turn scores. Instruct points concentrate top-right; the base chat-template condition supplies most low-score points across the whole diversity range; base raw multi-turn points cluster near a distinct 3-gram rate of 1.0 while spanning the full score range — diverse text that still scores low.

The distinct 3-gram companion separates failure modes: it correlates positively with judge score in the single-turn and instruct multi-turn conditions (rank correlation 0.14-0.38, p at most 0.05, n 125-200) but not in the base multi-turn condition (−0.06, p 0.49, n 125), where texts are lexically diverse (highest distinct 3-gram mean, 0.965) yet score low — so repetition loops are not what drives the low scores. The scatter does not itself identify the residual failure mode; the drift-and-inconsistency label comes from the judge rubric's low-score anchors (random topic jumps) and the sampled raw completions in Methodology, which show off-topic, self-inconsistent continuations.

Which render each existing base-row artifact actually used

Table-only result: these are provenance facts from the artifact inventory (2,846 files under the four lineage roots classified; 1,003 completion banks, 0 candidate base-chat banks, 0 indeterminate), with no quantity to plot.

No base-generated chat-template completion bank exists anywhere in the inventory, so nothing in the paper is invalidated by this verdict: the ~87% row is a teacher-forced fit whose validity this experiment does not measure either way, and the turn-dynamics base rows are already on the plain render. The restatement is editorial — state base single-turn map numbers on the plain-render values above and label the teacher-forced row as such.

Verifier-WARN acknowledgment: conciseness caps — several Takeaways bullets exceed the 30-word bullet cap, some sections exceed the 120-word result-prose band, and total content prose runs over the round-scaled 1,300-word total-prose budget; kept for numeric completeness of a folded two-round, nine-condition design; the provenance-table result is deliberately figure-less (provenance facts, no quantity to plot).

Tamer decoding lifts the chat render only to 56% coherent, while bare text clears the floor

What is plotted: fraction of 200 completions judged coherent per condition (bars, Wilson 95% intervals; per-item mean judge scores overlaid for the fresh conditions, right axis) for the four fresh decoding conditions and the two parent temperature-1.0 comparators; dashed line: the 0.80 decision floor.

Figure. Tamer decoding does not rescue the chat render. Both chat-template conditions stay far below the 0.80 floor at temperature 0.7 and under greedy decoding, while both bare-text conditions clear it; the temperature-1.0 comparators sit lower on both renders.

Per-unit view: item-mean score histograms, four fresh conditions.

Figure. Per-unit companion to the aggregate bars above. The chat conditions keep a heavy sub-50 mass at tamer decoding; the bare conditions concentrate above 90.

Tamer decoding helps both renders — mean per-prompt gains of roughly 23 points on chat and 9 to 11 on bare vs temperature 1.0 (200 pairs each) — but the chat render still reaches only 56.0% and 56.5% coherent: the collapse is the render, not the sampling temperature.

The verdict survives the 0.70 and 0.90 floors, the drops-scored-incoherent variant, and both intrusion recounts (Methodology table). Bare text clears the floor at 82.5% and 85.5%, so the sub-floor 71% at temperature 1.0 was partly a temperature effect. Greedy chat runs to the 1,024-token cap on 39% of completions (repetition loops), against 9.5% for greedy bare text.

Repro: Driver scripts/issue2477_base_coherence.py at code SHA 3f07e928c1 (branch issue-2477; analysis phases unchanged since — a figure-title-only touch-up landed at c492f44932); parent-round plan v4 at the task's plans/v4.md; the decoding-sensitivity round ran under plan v5 (amendment), now at plans/plan.md. Compute: one H100 (pod-2477, RunPod eval intent) for the two fresh generation conditions — about 12 minutes wall (launch 01:31 UTC, uploads verified 01:43 UTC, 2026-08-23); judge = 260 pilot + 4,250 production Batch API calls; aggregation and figures on the VM (CPU, minutes). The same-issue follow-up round decoding-sensitivity added pod-2477-decsens (RunPod eval intent, 1× H100, about 25 minutes wall — roughly 0.5 GPU-hours) for the four fresh conditions, plus a 208-draw judge pilot and 4,000 production Batch API calls; aggregation and figures again on the VM.

• Verdict + per-item scores: eval_results/issue_2477/coherence_verdict.json; pilot gate report: eval_results/issue_2477/judge/pilot_report.json; per-draw judge raw: eval_results/issue_2477/judge/judge_raw_arm_instruct_chat.json, judge_raw_arm_base_chat.json, judge_raw_arm_base_bare.json, judge_raw_arm_base_rawmt.json, judge_raw_arm_instruct_rawmt.json (all five mirrored on HF under issue2477_base_coherence/judge_raw/ @ c1cdf4bb98669511c1154fb1fbb2c11a7e539adc).
• Inventory triple (regenerated after the round-2 classifier fix, commit c1c8abb0b6): eval_results/issue_2477/inventory_manifest.json, eval_results/issue_2477/format_inventory.json, eval_results/issue_2477/format_inventory.md.
• Sample manifests: eval_results/issue_2477/samples/sample_manifest.json (200 chat prompts + 125 multi-turn pairs, seed 2477); fresh completions: eval_results/issue_2477/fresh_completions/base_chat_seed42.jsonl, base_bare_seed42.jsonl (canonical copies + gen_meta.json on HF: issue2477_base_coherence — listing verified live via the Hub API this round, 20 files).
• Prompt-script × response-script cross-tab (zero-GPU follow-up round): eval_results/issue_2477/lang_crosstab/lang_crosstab.json + generator scripts/issue2477_lang_crosstab.py, both at 7c87ab6c24 (branch issue-2477).
• Reused completion banks from #825: issue825_userbase_map/raw_completions/track_s/track_s.jsonl @ c1cdf4bb98669511c1154fb1fbb2c11a7e539adc (instruct chat-template comparator; 200 of its 5,000 rows) and issue825_userbase_map/raw_completions/turn_dynamics/armG/pretrained/shard0of3/ @ c1cdf4bb98669511c1154fb1fbb2c11a7e539adc + .../armG/instruct/shard0of3/ @ c1cdf4bb98669511c1154fb1fbb2c11a7e539adc (raw multi-turn conditions; 125 matched keys) — fit: same prompt panel / matched keys and the same sampling recipe as the fresh conditions; renders verified from track_s_meta.json and the per-model rollout fingerprints.
• Figures: figures/issue_2477/ (six PNG+PDF+meta triples) — five pinned at 3f07e928c1; the cap-hit triple re-rendered with a reader-facing title and pinned at c492f44932 (same data, title-only change).
• Decoding-sensitivity round artifacts (branch issue-2477 @ 9a3e7fe206): verdict + per-item scores eval_results/issue_2477/decoding-sensitivity/coherence_verdict.json; per-draw judge raw + pilot report under eval_results/issue_2477/decoding-sensitivity/judge/; fresh completions + gen_meta.json under eval_results/issue_2477/decoding-sensitivity/fresh_completions/; figures figures/issue_2477/ decoding_sensitivity_* (six PNG+PDF+meta triples, same pin). HF mirrors, listing verified live via the Hub API this round: raw_completions/generation/decoding_sensitivity (5 files) and judge_raw/decoding_sensitivity (9 files: 4 per-arm judge raws, the pilot report, and 4 per-arm pilot raws under pilot_raw/). Round condition slugs arm_base_chat_t07, arm_base_chat_t00, arm_base_bare_t07, arm_base_bare_t00; round verdict token render-driven; generation at commit 84452e335b, aggregation at 81a5fcd2dc (both on branch issue-2477).
• Judge model claude-sonnet-4-5-20250929; condition slugs arm_instruct_chat, arm_base_chat, arm_base_bare, arm_base_rawmt, arm_instruct_rawmt; verdict token restate-on-bare-text.
• Open code-hardening concerns from review (no data impact): the paid-phase skip guard's strict-boolean tightening is deferred to the next driver touch, and entry-guard regression tests remain a nit; the inventory-classifier close condition (diagnostics exclusion + regenerated triple before quoting per-class counts) was met at c1c8abb0b6.

Context: originating prompt (verbatim): Paper outline 2026-08-22: 'TO VERIFY: BASE MODEL COMPLETIONS IN CHAT TEMPLATE ARE COHERENT — ELSE USE BARE TEXT FORMAT'. Parent: #825 (context-to-answer map line). Kind: analysis (paper base-row format check). Run: 2026-08-22 → 2026-08-23; judge wave completed 02:30 UTC 2026-08-23. Same-issue follow-up round decoding-sensitivity (source: proposer cheap band; run 2026-08-23; plan v5), round prompt (verbatim excerpt from the proposer proposal):

Decoding-sensitivity re-measurement of the coherence verdict — Type: Ablation ... Hypothesis: The chat-template collapse is render-driven, not a temperature-1.0 sampling artifact: base-under-chat stays far below the 0.80 floor at temperature 0.7 and greedy (0.0), while base-under-bare-text improves from 0.71 toward or past 0.80. ... Differs from parent: Sampling temperature ONLY — 1.0 → {0.7, 0.0} (two levels of the one variable), crossed with the two existing fresh renders on the same 200 prompts.