More Thinking, More Bias: Length-Driven Position Bias in Reasoning Models
Authors: Xiao Wang
Summary
The paper challenges the assumption that chain-of-thought reasoning reduces shallow biases. Testing position bias (models favoring option A or D in multiple-choice questions), the authors find the opposite: longer reasoning trajectories correlate with stronger position bias within any reasoning-capable model. Across thirteen reasoning configurations (R1-distilled models, base models with CoT prompting, DeepSeek-R1 at 671B) on MMLU, ARC-Challenge, and GPQA, twelve show positive partial correlation between trajectory length and position bias (0.11–0.41, all p<0.05). A truncation experiment confirms causality: resuming reasoning from later points increases the shift toward position-preferred options. At 671B, aggregate bias is near zero, but the longest-trajectory quartile still shows the effect, suggesting accuracy gates the expression of length-driven bias rather than removing the mechanism.
Main takeaways:
- More reasoning (longer trajectories) → more position bias, not less, within reasoning-capable models.
- Twelve of thirteen reasoning configurations show significant positive correlation between trajectory length and position bias (0.11–0.41).
- Truncation experiment: resuming reasoning from later points causally increases position-biased shifts (16%–32% for R1-Qwen-7B).
- At 671B scale, aggregate bias is low (0.019) but the longest quartile still shows the effect (0.071)—accuracy suppresses but doesn't eliminate the mechanism.
- CoT reasoning replaces baseline direct-answer bias with a new length-accumulated bias; reasoning models should not be treated as order-robust in MCQ evaluation.
Relevance
Potentially relevant to my work on conditional behavior and persona installation. If reasoning length modulates bias expression, that's another axis along which implanted behaviors (like persona markers) might interact with model output. The finding that longer forward passes accumulate bias could inform how I interpret leakage experiments—perhaps longer system prompts or more reasoning steps change not just what's implanted but how it's expressed.
Abstract
arXiv:2605.06672v1 Announce Type: new Abstract: Chain-of-thought (CoT) reasoning and reasoning-tuned models such as DeepSeek-R1 are commonly assumed to reduce shallow heuristic biases by thinking carefully. We test this on position bias in multiple-choice QA and find a different story: within any reasoning-capable model, per-question position bias scales with the length of the reasoning trajectory. Across thirteen reasoning-mode configurations (two R1-distilled 7-8B models, two base models prompted with CoT, and DeepSeek-R1 at 671B) on MMLU, ARC-Challenge, and GPQA, twelve show a positive partial correlation between trajectory length and Position Bias Score (PBS) after controlling for accuracy, ranging from 0.11 to 0.41 (all p < 0.05). All twelve open-weight reasoning-mode configurations show monotonically increasing PBS across length quartiles. A truncation intervention provides causal evidence: continuations resumed from later points in the trajectory are increasingly likely to shift toward position-preferred options (16% to 32% for R1-Qwen-7B across absolute-position buckets). At 671B, aggregate PBS collapses to 0.019, but the length effect still manifests in the longest quartile (PBS = 0.071), suggesting that accuracy gates the expression of length-driven bias rather than eliminating the underlying mechanism. We additionally find that direct-answer position bias is a distinct phenomenon with a different footprint (strong in Llama-Instruct-direct, weak in Qwen-Instruct-direct, and uncorrelated with trajectory length): CoT reasoning replaces this baseline bias with length-accumulated bias. Our results argue that reasoning-capable models should not be treated as order-robust by default in MCQ evaluation pipelines, and offer a diagnostic toolkit (PBS, commitment change point, effective switching, truncation probes) for auditing position bias in reasoning models.