Reinforcement Learning from Human Feedback (RLHF) involves training policy models (PMs) and reward models (RMs) to align language models with human preferences. Instead of focusing solely on PMs and RMs independently, we propose to examine their interactions during fine-tuning, introducing the concept of seamlessness. Our study starts with observing the saturation phenomenon, where continual improvements in RM and PM do not translate into RLHF progress. Our analysis shows that RMs fail to assign proper scores to PM responses, resulting in a 35% mismatch rate with human preferences, highlighting a significant discrepancy between PM and RM. To measure seamlessness between PM and RM without human effort, we propose an automatic metric, SEAM. SEAM quantifies the discrepancies between PM and RM judgments induced by data samples. We validate the effectiveness of SEAM in data selection and model augmentation. Our experiments demonstrate that (1) using SEAM-filtered data for RL training improves RLHF performance by 4.5%, and (2) SEAM-guided model augmentation results in a 4% performance improvement over standard augmentation methods.

借助强化学习从人类反馈中进行训练，通过训练策略模型和奖励模型来使语言模型与人类偏好相一致；我们提出了研究对策略模型和奖励模型之间的交互作用进行微调的无缝度概念，探索了其对性能的影响，并引入了自动度量标准SEAM来度量两者之间的无缝度。实验证明，利用SEAM进行数据选择和模型增强可以显著提高强化学习从人类反馈中的性能。

奖励和策略模型在强化学习中的无缝衔接探讨