Mitigating Cognitive Bias in RLHF by Altering Rationality
Authors: Tiffany Horter, Andrew Markham, Niki Trigoni et al.
Summary
The authors address a limitation in RLHF: the reward model typically assumes a fixed "rationality" parameter (beta) that governs how consistently human preferences reflect true reward differences, but real human feedback is biased in context-dependent ways. They propose dynamically adjusting beta during reward learning using an LLM-as-judge to detect likely cognitive biases in each annotation, effectively downweighting comparisons that are probably unreliable or biased. Empirically, this produces a more rational downstream model even when training on heavily biased preference data.
Main takeaways:
- Standard RLHF uses a Boltzmann model with a fixed rationality parameter (beta) that assumes uniform annotator reliability, but human judgments are shaped by context-dependent cognitive biases
- The authors treat rationality as context- and annotation-dependent rather than fixed
- An LLM-as-judge assesses each preference comparison for likely cognitive bias, and beta is dynamically adjusted to downweight biased comparisons during reward learning
- This approach learns a more rational downstream model even when training on datasets with strongly biased preferences
- Addresses systematic deviations from reward-consistent behavior that arise contextually in human feedback
Relevance
Connects to my work on conditional behavior: this is about how human preferences are conditioned on context (biased in systematic, context-dependent ways), and the solution is to model that conditioning explicitly. While my focus is on how model behaviors are conditioned on personas/prompts/markers during training, the broader theme of context-dependent behavior and how to handle it is conceptually related.
Abstract
arXiv:2605.06895v1 Announce Type: new Abstract: How can we make models robust to even imperfect human feedback? In reinforcement learning from human feedback (RLHF), human preferences over model outputs are used to train a reward model that assigns scalar values to responses. Because these rewards are inferred from pairwise comparisons, this learning depends on an assumed relationship between latent reward differences and observed preferences, typically modeled using a Boltzmann formulation in which a rationality parameter beta informs how consistently preferences reflect reward differences. In practice, beta is typically treated as a fixed constant that reflects assumed uniform annotator reliability. However, human feedback is not this simplistic in practice: real human judgments are shaped by cognitive biases, leading to systematic deviations from reward-consistent behavior that arise contextually. To address this, we treat rationality as context- and annotation-dependent. We design an approach to dynamically adjust the rationality parameter beta during reward learning using an LLM-as-judge to assess the likely presence of cognitive biases. This approach effectively downweights comparisons that are likely to reflect biased or unreliable judgments. Empirically, we show that this approach learns a more rational downstream model, even when finetuning on datasets with strongly biased preferences.