Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences. Unlike fine-tuning, for which there are many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment process. Understanding this relationship is important as real user data may be collected and used to align large models; if user data is memorized during RLHF and later regurgitated, this could raise privacy concerns. In this work, we analyze how training data memorization can surface and propagate through each phase of RLHF. We focus our study on code completion models, as code completion is one of the most popular use cases for large language models. We find that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized, in comparison to aligning via directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF.

通过分析训练数据记忆在强化学习过程中如何表现和传播的方式，研究发现强化学习与人类反馈对齐方式相比直接微调数据对齐方式，更少地导致训练数据的记忆，但已经在微调阶段记忆的样本在RLHF过程中仍然保持记忆的情况居多，这对于保护隐私可能会带来潜在风险。

衡量RLHF中的代码完成功能的记忆化