The performance of the reward model (RM) is a critical factor in improving
the effectiveness of the large language model (LLM) during alignment
fine-tuning. There remain two challenges in RM training: 1) training the same
RM using various categories of data may cause its generalization performance to
suffer from multi-task disturbance, and 2) the human annotation consistency
rate is generally only $60\%$ to $75\%$, causing training data to contain a lot
of noise. To tackle these two challenges, we introduced the idea of
Mixture-of-Experts (MoE) into the field of RM for the first time. We propose
the Double-Layer MoE RM (DMoERM). The outer layer MoE is a sparse model. After
classifying an input into task categories, we route it to the corresponding
inner layer task-specific model. The inner layer MoE is a dense model. We
decompose the specific task into multiple capability dimensions and
individually fine-tune a LoRA expert on each one. Their outputs are then
synthesized by an MLP to compute the final rewards. To minimize costs, we call
a public LLM API to obtain the capability preference labels. The validation on
manually labeled datasets confirms that our model attains superior consistency
with human preference and outstrips advanced generative approaches. Meanwhile,
through BoN sampling and RL experiments, we demonstrate that our model
outperforms state-of-the-art ensemble methods of RM and mitigates the
overoptimization problem. Our code and dataset are available at:
this https URL

通过在奖励模型中引入 Mixture-of-Experts (MoE) 思想，我们提出了一种新的 Double-Layer MoE RM (DMoERM) 模型，该模型在任务分类和能力维度上的精细调优方面表现出卓越的性能，并超越了先进的生成方法。