Direct alignment methods are increasingly used for aligning large language models (LLMs) with human preferences. However, these methods suffer from the issues of verbosity and likelihood displacement, which can be driven by the noisy preference pairs that induce similar likelihood for preferred and dispreferred responses. The contributions of this paper are two-fold. First, we propose a new preference alignment method based on comparison oracles and provide the convergence guarantee for its basic scheme. Second, we improve our method using some heuristics and conduct the experiments to demonstrate the flexibility and compatibility of practical scheme in improving the performance of LLMs using noisy preference pairs. Evaluations are conducted across multiple base and instruction-tuned models (Mistral-7B, Llama-3-8B and Gemma-2-9B) with benchmarks (AlpacaEval 2, MT-Bench and Arena-Hard). Experimental results show the effectiveness of our method as an alternative to addressing the limitations of existing direct alignment methods. A highlight of our work is that we evidence the importance of designing specialized methods for preference pairs with distinct likelihood margin, which complements the recent findings in \citet{Razin-2025-Unintentional}.

本文解决了现有直接对齐方法在处理语言模型与人类偏好对齐中存在的冗长性和可能性偏移问题。提出了一种基于比较 oracle 的新偏好对齐方法，并通过实验展示了其在利用噪声偏好对进行语言模型性能提升中的灵活性和兼容性。实验结果证明了该方法作为替代方案的有效性，并强调了设计针对具有不同可能性边际的偏好对的专用方法的重要性。

ComPO：通过比较 oracle 实现偏好对齐