Reinforcement learning from human feedback (RLHF) has demonstrated great
promise in aligning large language models (LLMs) with human preference.
Depending on the availability of preference data, both online and offline RLHF
are active areas of investigation. A key bottleneck is understanding how to
incorporate uncertainty estimation in the reward function learned from the
preference data for RLHF, regardless of how the preference data is collected.
While the principles of optimism or pessimism under uncertainty are
well-established in standard reinforcement learning (RL), a
practically-implementable and theoretically-grounded form amenable to large
language models is not yet available, as standard techniques for constructing
confidence intervals become intractable under arbitrary policy
parameterizations.
In this paper, we introduce a unified approach to online and offline RLHF --
value-incentivized preference optimization (VPO) -- which regularizes the
maximum-likelihood estimate of the reward function with the corresponding value
function, modulated by a $\textit{sign}$ to indicate whether the optimism or
pessimism is chosen. VPO also directly optimizes the policy with implicit
reward modeling, and therefore shares a simpler RLHF pipeline similar to direct
preference optimization. Theoretical guarantees of VPO are provided for both
online and offline settings, matching the rates of their standard RL
counterparts. Moreover, experiments on text summarization and dialog verify the
practicality and effectiveness of VPO.

人类反馈引导的强化学习对齐大型语言模型的主要研究领域之一，本文通过引入一种统一的在线和离线强化学习方法 —— 以价值激励的偏好优化（VPO），实现了对奖励函数的不确定性估计，并在文本摘要和对话等任务上进行了实验证实其实用性和有效性。