$ξ$-DPO: Direct Preference Optimization via Ratio Reward Margin
Authors: Zhengyuan Fan, Zhonghua Wu, Yuxuan Du et al.
Summary
The authors propose ξ-DPO, a variant of preference optimization (fine-tuning LLMs from human preference data) that simplifies hyperparameter tuning. Existing methods like SimPO have two coupled hyperparameters (β and γ) that are hard to tune jointly because the margin γ doesn't have a consistent interpretation across datasets with different reward distributions. ξ-DPO reformulates the objective to use a "ratio reward margin" — the ratio of chosen to rejected response probabilities — which is bounded and interpretable, and can be set directly from the initial reward gap distribution without trial-and-error.
Main takeaways:
- SimPO's margin hyperparameter γ is not easily interpretable across datasets because it depends on the reward gap structure; tuning β and γ jointly is difficult.
- ξ-DPO redefines the reward as a ratio (chosen/rejected) rather than a difference, yielding a bounded margin ξ that explicitly represents desired relative separation.
- The ratio formulation cancels the effect of β, eliminating one hyperparameter from the tuning problem.
- ξ can be determined from the initial reward gap distribution, avoiding repeated trial-and-error tuning across datasets.
- Reformulates the preference objective as minimizing distance between reward gaps and optimal margins rather than maximizing likelihood of reward gaps.
Relevance
Directly relevant to my midtraining-stage behavioral installation and installation-path equivalence projects — this is about fine-tuning methods for installing behaviors (preferences) into LLMs, and understanding how hyperparameters control what gets installed is central to comparing fine-tuning as an installation path vs. prompting or steering.
Abstract
arXiv:2605.10981v1 Announce Type: new Abstract: Reference-free preference optimization has emerged as an efficient alternative to reinforcement learning from human feedback, with Simple Preference Optimization(SimPO) demonstrating strong performance by eliminating the explicit reference model through a simple objective. However, the joint tuning of the hyperparameters $\beta$ and $\gamma$ in SimPO remains a central challenge. We argue that this difficulty arises because the margin formulation in SimPO is not easily interpretable across datasets with different reward gap structures. To better understand this issue, we conduct a comprehensive analysis of SimPO and find that $\beta$ implicitly controls sample filtering, while the effect of $\gamma$ depends on the reward gap structure of the dataset. Motivated by these observations, we propose $\xi$-DPO: Direct preference optimization via ratio reward margin. We first reformulate the preference objective through an equivalent transformation, changing the optimization target from maximizing the likelihood of reward gaps to minimizing the distance between reward gaps and optimal margins. Then, we redefine the reward in a ratio form between the chosen and rejected, which effectively cancels the effect of $\beta$ and yields a bounded and interpretable margin. This margin is called the ratio reward margin and is denoted by $\xi$. Unlike the margin $\gamma$ in SimPO, $\xi$ explicitly represents the desired relative separation between chosen and rejected responses and can be determined from the initial reward gap distribution, avoiding repeated trial-and-error tuning. ....