Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs

Authors: Amin Banayeeanzade, Qingchuan Yang, Dhruv Tarsadiya et al.

arXiv · PDF

Summary

The authors diagnose why LLMs collapse into repetitive, low-diversity outputs even when many valid continuations exist. They introduce a "validity-diversity" framework showing that the problem stems from two forms of miscalibration: (1) valid tokens aren't reliably ranked above invalid ones (order calibration), so sampling methods must trade validity for diversity, and (2) probability mass is overly concentrated on a few valid options with a long tail of mixed valid/invalid tokens (shape calibration). These local failures compound across decoding steps, sharply reducing sequence-level diversity. Experiments across 14 models confirm the pattern.

Main takeaways:

• Diversity collapse isn't just a sampling algorithm problem—it's baked into LLM probability distributions via miscalibration
• Order calibration failure: valid tokens don't consistently rank above invalid ones, forcing validity-diversity tradeoffs
• Shape calibration failure: probability mass is spiked on few valid options, with a heavy tail mixing valid and invalid
• Local miscalibration at each step compounds across decoding, producing large sequence-level diversity losses
• Observed across 14 models of different families and scales, suggesting a fundamental distribution problem

Relevance

Tangentially relevant if I explore how persona/marker installation changes output distributions: this framework for decomposing probability-mass allocation (order vs shape calibration) could be useful for understanding whether marker training changes which completions are possible (shape) or just their ranking (order).

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure, failures, limitation, benchmark.

Abstract

arXiv:2605.11128v1 Announce Type: new Abstract: Diversity is essential for language-model applications ranging from creative generation to scientific discovery, yet modern LLMs often collapse into a narrow subset of plausible outputs. While prior work has developed benchmarks for measuring this lack of diversity, less is known about how the step-by-step probability distributions at inference time cause the problem. We introduce a validity--diversity framework that attributes diversity collapse to how an LLM allocates probability mass across valid and invalid continuations during decoding. This framework decomposes the bottleneck into two complementary forms of miscalibration. First, order calibration: valid tokens are not reliably ranked above invalid tokens, so rank-based cutoff rules must trade off between recovering valid continuations and admitting invalid ones. Second, shape calibration: probability mass is overly concentrated only on few valid continuations while having a heavy-tail of mixed valid and invalid tokens, so maintaining high validity limits diversity. We formalize both mechanisms and show that local failures compound across decoding steps, producing strong sequence-level losses in diversity. Empirically, we develop controlled diagnostics for probing these bottlenecks, including tasks with exactly known valid sets and oracle cutoff baselines. Across 14 language models spanning multiple families and scales, we find that diversity collapse is not merely a limitation of particular sampling heuristics, but a consequence of order and shape miscalibration in the LLM distribution.