Reinforcement learning from human feedback (RLHF) has emerged as the primary
method for aligning large language models (LLMs) with human preferences. The
RLHF process typically starts by training a reward model (RM) using human
preference data. Conventional RMs are trained on pairwise responses to the same
user request, with relative ratings indicating which response humans prefer.
The trained RM serves as a proxy for human preferences. However, due to the
black-box nature of RMs, their outputs lack interpretability, as humans cannot
intuitively understand why an RM thinks a response is good or not. As RMs act
as human preference proxies, we believe they should be human-interpretable to
ensure that their internal decision processes are consistent with human
preferences and to prevent reward hacking in LLM alignment. To build RMs with
interpretable preferences, we propose a two-stage approach: i) train an
Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional
absolute-rating data, each dimension corresponding to a human-interpretable
objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts
(MoE) strategy with a gating network that automatically selects the most
suitable reward objectives based on the context. We efficiently trained an
ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top
of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art
performance on RewardBench, a benchmark evaluating RMs for language modeling.
Notably, the performance of our model surpasses the LLM-as-a-judge method with
GPT-4 judges by a margin, and approaches the performance of the much larger
Nemotron-4 340B reward model.

通过建立一个具有可解释性的绝对评分多目标奖励模型 (ArmoRM) 和一个自动选择最合适奖励目标的门控网络，我们在大型语言模型对齐领域取得了与 GPT-4 评委相比的最先进表现，并接近更大的 Nemotron-4 340B 奖励模型的性能。