Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged
as an effective approach for scaling Transformer models. However, they often
suffer from computational inefficiency since a significant number of parameters
are unnecessarily involved in computations via multiplying values by zero or
low activation values. To address this issue, we present \tool, a novel MoE
designed to enhance both the efficacy and efficiency of sparse MoE models.
\tool leverages small experts and a threshold-based router to enable tokens to
selectively engage only essential parameters. Our extensive experiments on
language modeling and machine translation tasks demonstrate that \tool can
enhance model performance while decreasing the computation load at MoE layers
by over 50\% without sacrificing performance. Furthermore, we present the
versatility of \tool by applying it to dense models, enabling sparse
computation during inference. We provide a comprehensive analysis and make our
code available at this https URL

提出了	ool，一种新颖的 MoE 模型，通过利用小型专家和基于阈值的路由器，实现了对模型性能的提升，并在减少计算负载 50% 以上的同时，不牺牲性能。