Reward models (RMs) are at the crux of successful RLHF to align pretrained
models to human preferences, yet there has been relatively little study that
focuses on evaluation of those reward models. Evaluating reward models presents
an opportunity to understand the opaque technologies used for alignment of
language models and which values are embedded in them. To date, very few
descriptors of capabilities, training methods, or open-source reward models
exist. In this paper, we present RewardBench, a benchmark dataset and code-base
for evaluation, to enhance scientific understanding of reward models. The
RewardBench dataset is a collection of prompt-win-lose trios spanning chat,
reasoning, and safety, to benchmark how reward models perform on challenging,
structured and out-of-distribution queries. We created specific comparison
datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect
facts) why one answer should be preferred to another. On the RewardBench
leaderboard, we evaluate reward models trained with a variety of methods, such
as the direct MLE training of classifiers and the implicit reward modeling of
Direct Preference Optimization (DPO), and on a spectrum of datasets. We present
many findings on propensity for refusals, reasoning limitations, and
instruction following shortcomings of various reward models towards a better
understanding of the RLHF process.

RewardBench 是一个用于评估奖励模型的基准数据集和代码库，旨在增强对奖励模型科学理解的同时，深入了解用于语言模型对齐的不透明技术以及其中嵌入的价值观。通过包含多个难度级别、结构化和分布外查询的提示 - 赢 - 输三元组，我们对通过各种方法训练的奖励模型进行了评估，并针对拒绝、推理限制和指示跟随等方面的问题提出了许多发现，以促进对 RLHF 过程的更好理解。