Variance-aware Reward Modeling with Anchor Guidance

Authors: Shuxing Fang, Ruijian Han, Liangyu Zhang et al.

arXiv · PDF

Summary

The authors tackle reward modeling when human preferences are pluralistic (people disagree). Standard Bradley-Terry models can only capture disagreement by shrinking reward margins, and Gaussian reward models (which predict both mean and variance) are fundamentally non-identifiable from pairwise comparisons alone. They propose augmenting preference data with two coarse "anchor" labels per response (e.g., "good" vs. "bad") to resolve this non-identifiability, prove that two anchors are sufficient, and show both theoretically and empirically that the method improves reward modeling and downstream RLHF (PPO and best-of-N).

Main takeaways:

• Standard Bradley-Terry reward models can't properly represent disagreement—they just shrink reward gaps when preferences are noisy.
• Gaussian reward models (predicting mean and variance) are better in principle but suffer from non-identifiability: you can't uniquely recover the variance from pairwise preferences alone.
• Adding two anchor labels (coarse quality scores like "good" or "bad") per response resolves the identifiability problem.
• The authors prove that two anchors are mathematically sufficient, develop a joint training objective, and establish convergence rates for both mean and variance.
• Experiments on four diverging-preference datasets show consistent improvements in reward modeling, PPO training, and best-of-N selection.

Relevance

Relevant to understanding how behavioral alignment is installed during RLHF—this is about the reward modeling step. The variance-aware approach could connect to my work if I'm thinking about how fine-tuning installs behaviors differently depending on training-data disagreement or noise.

Abstract

arXiv:2605.11865v1 Announce Type: new Abstract: Standard Bradley--Terry (BT) reward models are limited when human preferences are pluralistic. Although soft preference labels preserve disagreement information, BT can only express it by shrinking reward margins. Gaussian reward models provide an alternative by jointly predicting a reward mean and a reward variance, but suffer from a fundamental non-identifiability from pairwise preferences alone. We propose Anchor-guided Variance-aware Reward Modeling, a framework that resolves this non-identifiability by augmenting preference data with two coarse response-level anchor labels. Building on this, we prove that two anchors are sufficient for identification, develop a joint training objective and establish a non-asymptotic convergence rate for both the estimated reward mean and variance functions. Across simulation studies and four real-world diverging-preference datasets, our method consistently improves reward modeling performance and downstream RLHF, including PPO training and best-of-$N$ selection.