Aligning large language models (LLMs) with human preferences is critical for real-world deployment, yet existing methods like RLHF face computational and stability challenges. While DPO establishes an offline paradigm with single hyperparameter $\beta$, subsequent methods like SimPO reintroduce complexity through dual parameters ($\beta$, $\gamma$). We propose {ReLU-based Preference Optimization (RePO)}, a streamlined algorithm that eliminates $\beta$ via two advances: (1) retaining SimPO's reference-free margins but removing $\beta$ through gradient analysis, and (2) adopting a ReLU-based max-margin loss that naturally filters trivial pairs. Theoretically, RePO is characterized as SimPO's limiting case ($\beta \to \infty$), where the logistic weighting collapses to binary thresholding, forming a convex envelope of the 0-1 loss. Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO and SimPO across multiple base models, requiring only one hyperparameter to tune.

本研究解决了将大型语言模型与人类偏好对齐过程中面临的计算和稳定性挑战。提出了一种新的ReLU-based Preference Optimization (RePO)算法，简化了调参过程，通过保留SimPO的无参考边界并去除复杂的超参数，显著提升了模型性能。实验证明，RePO在多个基础模型上超越了现有方法DPO和SimPO，证明了其在优化极限情况下的有效性。

RePO: 基于ReLU的偏好优化