Large language models (LLMs) have achieved remarkable success, yet aligning their generations with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit or implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences using discrete latent codes. LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three alignment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-8B, and Llama3-8B-Instruct). Furthermore, deeper analysis reveals that the learned latent codes effectively capture the differences in the distribution of human preferences and significantly enhance the robustness of alignment against noise in data. By providing a unified representation for the multifarious preference factors, LPC paves the way towards developing more robust and versatile alignment techniques for the responsible deployment of powerful LLMs.

本研究解决了大型语言模型（LLMs）与人类偏好对齐的挑战，现有的偏好建模方法通常忽视了人类偏好的复杂性和多样性。我们提出了潜在偏好编码（LPC），该框架利用离散潜在编码自动推断数据中的隐性因素及其重要性，超越了传统的奖励函数依赖。实验表明，LPC在多项基准上优于现有对齐算法，并显著提升了对噪声数据的对齐鲁棒性，为更可靠的LLM对齐技术奠定基础。

潜在偏好编码：通过离散潜在编码对大型语言模型进行对齐