Reward models are critical in techniques like Reinforcement Learning from Human Feedback (RLHF) and Inference Scaling Laws, where they guide language model alignment and select optimal responses. Despite their importance, existing reward model benchmarks often evaluate models by asking them to distinguish between responses generated by models of varying power. However, this approach fails to assess reward models on subtle but critical content changes and variations in style, resulting in a low correlation with policy model performance. To this end, we introduce RM-Bench, a novel benchmark designed to evaluate reward models based on their sensitivity to subtle content differences and resistance to style biases. Extensive experiments demonstrate that RM-Bench strongly correlates with policy model performance, making it a reliable reference for selecting reward models to align language models effectively. We evaluate nearly 40 reward models on RM-Bench. Our results reveal that even state-of-the-art models achieve an average performance of only 46.6%, which falls short of random-level accuracy (50%) when faced with style bias interference. These findings highlight the significant room for improvement in current reward models. Related code and data are available at https://github.com/THU-KEG/RM-Bench.

本研究针对当前奖励模型基准测试中未能有效评估细微内容变化和风格偏见的问题，提出了RM-Bench这一新基准。通过对近40个奖励模型的评估，结果显示当前先进模型在面对风格偏见干扰时，仅达到46.6%的平均性能，表明奖励模型的改进空间巨大，提供了一个可靠的选择参考以提升语言模型的对齐效率。

RM-Bench：语言模型奖励模型的细微差别和风格基准测试