---

title: Persona-style rationale does not reduce answer uncertainty below generic rationale after answer-cue filtering (HIGH confidence) kind: experiment tags:

• todo
• mentor-followup created_at: '2026-05-11T23:32:14.000Z' has_clean_result: true sagan_id: edea817f-1c24-4fe2-8160-8bf3e8ee8b69 sagan_number: 355 priority: normal

---

Persona-style rationale does not reduce answer uncertainty below generic rationale after answer-cue filtering (HIGH confidence)

TL;DR

• Motivation: This follow-up to #186 asks whether the written rationale itself carries the wrong final answer, or whether the later answer decode still adds an important mechanism.
• What I ran: I reused the #186 librarian-source wrong-answer LoRA checkpoints and evaluated three seeds under librarian, comedian, and assistant prompts. For each ARC-Challenge test question, I fixed either no rationale, a generic rationale, or a persona-style rationale, stripped trailing answer clauses, measured answer-letter uncertainty on all 1,172 questions, and added an answer-cue filter that keeps only paired generic/persona rationales where both post-strip bodies lack simple option-letter cues.
• Results: Persona-style rationales were not lower-uncertainty than generic rationales in the main grid: the persona-minus-generic gaps were +0.082 nats for librarian, +0.033 for comedian, and +0.106 for assistant. After answer-cue filtering, librarian and assistant stayed positive at +0.069 and +0.083 nats; comedian flipped slightly to -0.014 nats, so the all-prompt direction is not filter-robust figure below.
• Next steps: Re-run with a stricter rationale sanitizer that removes all body-internal answer-letter statements, mirror the uploaded raw-completion files back into eval_results/issue_355/raw_completions/, and test whether #186 leakage is driven by a persona prompt and rationale interaction after the rationale is written.

Figure

Caption: Bars show mean answer-letter uncertainty after no rationale, a generic rationale, or a persona-style rationale across three eval prompts; lower bars mean the next answer letter is more pinned, and seed error bars show that the generic/persona ordering is stable in the unfiltered main grid.

Details

I measured uncertainty over the next answer letter after the prompt already contained the question and, when applicable, a saved #186 rationale. The maximum random-over-four-letters value is 1.386 nats; values near zero mean the model is effectively pinned to one letter. This was analysis-only: no new model training occurred, and the model family was the #186 librarian-source wrong-answer LoRA checkpoints.

I had pre-committed to looking for a 0.5-nat gap in the predicted direction, persona-style lower than generic, in both the librarian and assistant-prompt baseline cells for HIGH-confidence support of the carrier claim. The observed gap is +0.03 to +0.11 nats in the opposite direction, an order of magnitude smaller.

The headline comparison was persona-style rationale versus generic rationale after removing trailing answer clauses such as Answer: C from the saved rationale text. Seed-averaged answer-letter uncertainty was:

Eval prompt	No rationale	Generic rationale	Persona-style rationale	Persona minus generic
Librarian	0.369	0.042	0.124	+0.082
Comedian	0.260	0.045	0.078	+0.033
Assistant	0.164	0.039	0.145	+0.106

A wider top-token entropy check mostly agreed with the main metric: librarian generic/persona was 0.936 versus 1.088 nats, while assistant generic/persona was nearly tied at 1.128 versus 1.123 nats. This supports the librarian finding but keeps the assistant result tied to the answer-letter metric.

The strip pipeline itself is asymmetric and can produce the observed sign without a real persona-content effect. Averaged over the nine seed-by-eval-prompt persona-style cells, rule 1 removed an Answer: X trailer about 1,153 times per 1,172-question cell; by eval prompt, the rule-1 totals were 3,494, 3,448, and 3,438 out of 3,516 rows. For generic rationales, rule 0 left the body unchanged about 581 times per 1,172-question cell; by eval prompt, the rule-0 totals were 1,554, 1,750, and 1,924 out of 3,516 rows. A fully stripped persona-style rationale leaves less answer-letter signal visible to the conditional decoder than a generic rationale whose final answer-like body remains intact, so this asymmetry predicts higher persona-style uncertainty independently of any persona-vs-generic content difference. The pre-renormalization A/B/C/D mass also points this way in the librarian and assistant prompts: generic/persona was 0.556 versus 0.498 for librarian and 0.465 versus 0.454 for assistant, with comedian as the exception at 0.488 versus 0.519.

I tested that confound directly with scripts/issue_355/strip_confound_filter.py. The filter keeps a question only if both the generic and persona-style post-strip bodies lack these case-insensitive patterns: option [A-D], ([A-D]), answer is [A-D], or [A-D] is correct. It then recomputes paired means on the retained question ids.

Eval prompt	Retained rows per seed	Unfiltered gap	Filtered gap
Librarian	519, 497, 569	+0.082	+0.069
Comedian	328, 330, 348	+0.033	-0.014
Assistant	330, 333, 367	+0.106	+0.083

The answer-cue filter does not preserve the sign in the comedian prompt, so I do not claim that every eval prompt survives the body-cue control. It does preserve positive gaps in the librarian and assistant cells named by the plan, which means the body-cue confound does not rescue the predicted persona-style-lower-than-generic direction there.

Prompt-tail samples are findable in eval_results/issue_355/smoke_prompts.json for q_ids 0, 6, and 9. All 1,172 local source rationales per seed and eval prompt are in eval_results/issue186/librarian_persona_cot_seed*/result.json, and the raw empirical completions are browsable at the Hugging Face data path linked below.

Prompt tail	q_id	Excerpt
Librarian generic, letter-clean body	0	Therefore, the most likely effect of the increase in rotation is that planetary days will become shorter.\nAnswer:
Librarian generic, residual answer phrase	6	Therefore, the most logical reason for this behavior is to store food that will be eaten over the winter months. Answer choice C is the most likely correct answer.\nAnswer:
Librarian generic, residual option token	9	Therefore, the correct answer is (A) the atom.\nAnswer:
Librarian persona-style, residual option phrase	0	making option D the most likely effect.\n</persona-thinking>\n</persona-thinking>\nAnswer:
Librarian persona-style, letter-clean body	6	the most likely reason for this behavior is to repare for migration before winter.\n</persona-thinking>\n</persona-thinking>\nAnswer:
Librarian persona-style, letter-clean body	9	atoms are the smallest units that make up copper and maintain its characteristics.\n</persona-thinking>\n</persona-thinking>\nAnswer:

The paired question-level diagnostic also went against the carrier prediction: all nine seed-by-eval-prompt comparisons had positive mean persona-minus-generic gaps, with corrected p-values below 1.1e-20 and 1,164 to 1,172 paired questions per comparison. The empirical eight-sample check was directionally useful but noisy at the question level; rank correlations between analytical answer-letter uncertainty and empirical sampled-answer uncertainty were positive in all nine cells.

Note: the bias-corrected empirical entropy estimate for seed 256 ran 4-6x higher than seeds 42 and 137 in every persona-by-rationale-style cell. This is likely sampling noise from 200 questions at temperature 1.0; it is a diagnostic anomaly that does not affect the analytical headline.

The cross-seed memorization check did not explain the result. For librarian persona-style rationales, cross-seed answer uncertainty was not meaningfully higher than within-seed uncertainty; the cross-seed gap of +0.003 nats was well within sampling noise, so memorization is not the driver.

The comedian-source confirmation matched the librarian-source direction on the unfiltered main metric. With comedian-source seed 42 rationales, the comedian eval prompt had persona-style uncertainty 0.0955 versus generic 0.0248, a +0.0707 nats gap; the matching librarian-source seed 42 gap was +0.0340 nats. The librarian eval prompt was also positive at +0.0692 nats, and the assistant prompt was positive but small at +0.0148 nats.

Confidence: HIGH - the implemented trailing-answer-stripped measurement reverses the predicted ordering in the librarian and assistant cells required by the plan, and those two cells remain positive after filtering out paired rationales with simple body-internal option-letter cues. The confidence does not extend to an all-prompt claim after filtering, because the comedian filtered subset is slightly negative.

Parameter	Value
Model family	#186 librarian-source wrong-answer LoRA checkpoints
Base model	Qwen2.5-7B-Instruct
Seeds	42, 137, 256
Eval prompts	librarian, comedian, assistant
Rationale styles	no rationale, generic rationale, persona-style rationale
Analytical sample	1,172 ARC-Challenge test questions per seed and condition
Empirical sample	200 stratified ARC-Challenge test questions, 8 samples each
Strip scope	trailing answer clauses only; answer-letter statements inside rationale bodies were left intact
Primary figure	artifacts/hero.png
Answer-cue filter JSON	eval_results/issue_355/strip_confound_filter.json

Reproducibility

Artifacts:

• Model: hf-hub
• Dataset: n/a (analysis-only on parent #186 LoRA)
• Raw completions: hf-hub
• WandB run: n/a (analysis-only, no training)
• Eval JSON: eval_results/issue_355/aggregate.json and eval_results/issue_355/strip_confound_filter.json

Compute: 16m 44s on 1x A100 80GB on pod-355 for the original entropy run; the answer-cue filter is local CPU-only.

Code: scripts/measure_cot_entropy.py, configs/eval/issue355_entropy.yaml, scripts/issue_355/compute_deferred_stats_and_plot.py, scripts/issue_355/strip_confound_filter.py.

Git commit: 04e042735456f72c597a91a48bf066d0823d4fb7.

git clone https://github.com/superkaiba/explore-persona-space.git
cd explore-persona-space
git checkout 04e042735456f72c597a91a48bf066d0823d4fb7
uv sync --locked
uv run python scripts/measure_cot_entropy.py --config-name issue355_entropy output_dir=eval_results/issue_355

UV_CACHE_DIR=/tmp/uv-cache uv run python scripts/issue_355/compute_deferred_stats_and_plot.py

UV_CACHE_DIR=/tmp/uv-cache uv run python scripts/issue_355/strip_confound_filter.py