Generative foundation models are susceptible to implicit biases that can
arise from extensive unsupervised training data. Such biases can produce
suboptimal samples, skewed outcomes, and unfairness, with potentially
significant repercussions. Consequently, aligning these models with human
ethics and preferences is an essential step toward ensuring their responsible
and effective deployment in real-world applications. Prior research has
primarily employed Reinforcement Learning from Human Feedback (RLHF) as a means
of addressing this problem, wherein generative models are fine-tuned using RL
algorithms guided by a human-feedback-informed reward model. However, the
inefficiencies and instabilities associated with RL algorithms frequently
present substantial obstacles to the successful alignment of generative models,
necessitating the development of a more robust and streamlined approach. To
this end, we introduce a new framework, Reward rAnked FineTuning (RAFT),
designed to align generative models more effectively. Utilizing a reward model
and a sufficient number of samples, our approach selects the high-quality
samples, discarding those that exhibit undesired behavior, and subsequently
assembles a streaming dataset. This dataset serves as the basis for aligning
the generative model and can be employed under both offline and online
settings. Notably, the sample generation process within RAFT is gradient-free,
rendering it compatible with black-box generators. Through extensive
experiments, we demonstrate that our proposed algorithm exhibits strong
performance in the context of both large language models and diffusion models.

本文提出了一种新的框架 RAFT，它利用奖励模型和足够数量的样本将生成模型对齐，选择高质量的样本并去除那些表现不良的样本。该算法在大型语言模型和扩散模型的情况下表现良好。