Aligning Large Language Models (LLMs) to cater to different human preferences, learning new skills, and unlearning harmful behavior is an important problem. Search-based methods, such as Best-of-N or Monte-Carlo Tree Search, are performant, but impractical for LLM adaptation due to their high inference cost. On the other hand, using Reinforcement Learning (RL) for adaptation is computationally efficient, but performs worse due to the optimization challenges in co-training the value function and the policy. We present a new framework for reward optimization, Value Augmented Sampling (VAS), that can maximize different reward functions using data sampled from only the initial, frozen LLM. VAS solves for the optimal reward-maximizing policy without co-training the policy and the value function, making the optimization stable, outperforming established baselines, such as PPO and DPO, on standard benchmarks, and achieving comparable results to Best-of-128 with lower inference cost. Unlike existing RL methods that require changing the weights of the LLM, VAS does not require access to the weights of the pre-trained LLM. Thus, it can even adapt LLMs (e.g., ChatGPT), which are available only as APIs. In addition, our algorithm unlocks the new capability of composing several rewards and controlling the extent of each one during deployment time, paving the road ahead for the future of aligned, personalized LLMs.

通过价值增强抽样（VAS）的奖励优化框架，不需要共同训练策略和值函数的情况下，最大化不同奖励函数，相较于现有基线模型，在标准基准测试中不仅优于 PPO 和 DPO，而且与 Best-of-128 相比具有更低的推理成本，从而实现了优化的稳定性，并能适应仅作为 API 提供的 LLMs（例如 ChatGPT），同时为对齐的个性化 LLMs 的未来铺平道路。

增值取样用于语言模型对齐和个性化