Reinforcement Learning from Human Feedback (RLHF) is the current dominating
framework to fine-tune large language models to better align with human
preferences. However, the underlying premise of algorithms developed under this
framework can be problematic when user preferences encoded in human feedback
are diverse. In this work, we aim to address this problem by developing methods
for building personalized language models. We first formally introduce the task
of learning from personalized human feedback and explain why vanilla RLHF can
be problematic in this context. We then propose a general Personalized-RLHF
(P-RLHF) framework, which requires one to jointly learn a user model and a
language (or reward) model. The user model takes in user information and
outputs user representations. Its structure encodes our assumptions about user
preferences underlying the feedback data. We develop new learning objectives
for personalized reward modeling and personalized Direct Preference
Optimization. To demonstrate the efficacy of our method, we test it on
real-world text summarization data with annotated preferences and annotator
information. We fine-tune GPT-J 6B to obtain personalized language (and reward)
models, which outperform non-personalized models in terms of aligning with
individual preferences.

发展个性化语言模型的方法，结合用户模型和语言（或奖励）模型的学习目标，对个性化语言模型进行强化学习，以更好地满足用户偏好。

个性化人类反馈的个性化语言建模

Personalized Language Modeling from Personalized Human Feedback

In this paper, we detail novel strategies for interpolating personalized
language models and methods to handle out-of-vocabulary (OOV) tokens to improve
personalized language models. Using publicly available data from Reddit, we
demonstrate improvements in offline metrics at the user level by interpolating
a global LSTM-based authoring model with a user-personalized n-gram model. By
optimizing this approach with a back-off to uniform OOV penalty and the
interpolation coefficient, we observe that over 80% of users receive a lift in
perplexity, with an average of 5.2% in perplexity lift per user. In doing this
research we extend previous work in building NLIs and improve the robustness of
metrics for downstream tasks.

本文论述了插值个性化语言模型的新策略和处理未登录词的方法，以提高个性化语言模型，并通过 Reddit 的公开数据演示了通过将全局基于 LSTM 的撰写模型与用户个性化 n-gram 模型插值，通过优化基于分布式的 OOV 惩罚和插值系数的后退方法，我们观察到超过 80％的用户的恶化在困惑度上得到了提升，平均每个用户困惑度提升 5.2％。通过该研究，我们扩展了以前建立 NLIs 的工作，并提高了用于下游任务的指标的鲁棒性。