Reinforcement Learning from Human Feedback (RLHF) has enabled significant
advancements within language modeling for powerful, instruction-following
models. However, the alignment of these models remains a pressing challenge as
the policy tends to overfit the learned ``proxy" reward model past an
inflection point of utility as measured by a ``gold" reward model that is more
performant -- a phenomenon known as \textit{over-optimization}. Prior work has
mitigated this issue by computing a pessimistic statistic over an ensemble of
reward models, which is common in Offline Reinforcement Learning but incredibly
costly for language models with high memory requirements, making such
approaches infeasible for sufficiently large models. To this end, we propose
using a shared encoder but separate linear heads. We find this leads to similar
performance as the full ensemble while allowing tremendous savings in memory
and time required for training for models of similar size. \end{abstract}

使用共享编码器但独立的线性头部，以减小存储和训练时间开销，解决了语言模型在强化学习中的过度优化问题。