This paper studies the theoretical framework of the alignment process of
generative models with Reinforcement Learning from Human Feedback (RLHF). We
consider a standard mathematical formulation, the reverse-KL regularized
contextual bandit for RLHF. Despite its widespread practical application, a
rigorous theoretical analysis of this formulation remains open. We investigate
its theoretical properties both in offline and online settings and propose
efficient algorithms with finite-sample theoretical guarantees. Our work
bridges the gap between theory and practice by linking our theoretical insights
with existing practical alignment algorithms such as Direct Preference
Optimization (DPO) and Rejection Sampling Optimization (RSO). Furthermore,
these findings and connections also offer both theoretical and practical
communities new tools and insights for future algorithmic design of alignment
algorithms.

研究生成模型与强化学习从人类反馈中的对齐过程的理论框架，考虑了逆 KL 正则化情境赌博机在此方面的应用，通过离线和在线设置的理论属性研究以及与现有实际对齐算法的联系，为未来算法设计提供新的工具和见解。