Mixture-of-Experts (MoE) has gained increasing popularity as a promising
framework for scaling up large language models (LLMs). However, the reliability
assessment of MoE lags behind its surging applications. Moreover, when
transferred to new domains such as in fine-tuning MoE models sometimes
underperform their dense counterparts. Motivated by the research gap and
counter-intuitive phenomenon, we propose $\texttt{MoE-RBench}$, the first
comprehensive assessment of SMoE reliability from three aspects: $\textit{(i)}$
safety and hallucination, $\textit{(ii)}$ resilience to adversarial attacks,
and $\textit{(iii)}$ out-of-distribution robustness. Extensive models and
datasets are tested to compare the MoE to dense networks from these reliability
dimensions. Our empirical observations suggest that with appropriate
hyperparameters, training recipes, and inference techniques, we can build the
MoE model more reliably than the dense LLM. In particular, we find that the
robustness of SMoE is sensitive to the basic training settings. We hope that
this study can provide deeper insights into how to adapt the pre-trained MoE
model to other tasks with higher-generation security, quality, and stability.
Codes are available at this https URL

通过 $	exttt {MoE-RBench}$ 全面评估了混合专家模型（MoE）在可靠性方面的表现，包括安全性与幻觉、对抗攻击的弹性和超域鲁棒性，并发现在适当的超参数、训练配方和推理技术下，MoE 模型的可靠性可以超过密集型大型语言模型（LLMs）。