Large language models~(LLMs) are expected to be helpful, harmless, and honest. In various alignment scenarios, such as general human preference, safety, and confidence alignment, binary preference data collection and reward modeling are resource-intensive but necessary for human preference transferring. In this work, we explore using the similarity between sampled generations and high-quality reference answers as an alternative reward function for LLM alignment. Using similarity as a reward circumvents training reward models, and collecting a single reference answer potentially costs less time than constructing binary preference pairs when multiple candidates are available. Specifically, we develop \textit{RefAlign}, a versatile REINFORCE-style alignment algorithm, which is free of reference and reward models. Instead, RefAlign utilizes BERTScore between sampled generations and high-quality reference answers as the surrogate reward. Beyond general human preference optimization, RefAlign can be readily extended to diverse scenarios, such as safety and confidence alignment, by incorporating the similarity reward with task-related objectives. In various scenarios, {RefAlign} demonstrates comparable performance to previous alignment methods while offering high efficiency.

本研究解决了大型语言模型（LLMs）对齐过程中对人类偏好数据收集的高成本问题。我们提出了一种新方法，利用样本生成与高质量参考答案之间的相似性作为奖励函数，通过开发RefAlign算法，显著提高了对齐效率，并适用于各种对齐场景，如安全和信任度对齐。研究结果显示，RefAlign在多个场景中具备与传统方法相当的性能，且更具高效性。

通过参考答案学习：无需二元人类偏好数据的多功能语言模型对齐