Understanding human preferences is crucial for improving foundation models and building personalized AI systems. However, preferences are inherently diverse and complex, making it difficult for traditional reward models to capture their full range. While fine-grained preference data can help, collecting it is expensive and hard to scale. In this paper, we introduce Decomposed Reward Models (DRMs), a novel approach that extracts diverse human preferences from binary comparisons without requiring fine-grained annotations. Our key insight is to represent human preferences as vectors and analyze them using Principal Component Analysis (PCA). By constructing a dataset of embedding differences between preferred and rejected responses, DRMs identify orthogonal basis vectors that capture distinct aspects of preference. These decomposed rewards can be flexibly combined to align with different user needs, offering an interpretable and scalable alternative to traditional reward models. We demonstrate that DRMs effectively extract meaningful preference dimensions (e.g., helpfulness, safety, humor) and adapt to new users without additional training. Our results highlight DRMs as a powerful framework for personalized and interpretable LLM alignment.

本研究解决了传统奖励模型难以捕捉多样化和复杂的人类偏好的问题。提出了一种新的“分解奖励模型”（DRMs），通过二元比较提取人类偏好而无需精细注释，并利用主成分分析（PCA）对偏好进行向量表示和分析。研究表明，DRMs能够有效提取有意义的偏好维度，并适应新用户，可能为个性化和可解释的语言模型对齐提供强有力的框架。

通过主成分分析重新思考多样化的人类偏好学习