Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, effective training of SMoE has proven to be challenging due to the representation collapse issue, which causes parameter redundancy and limited representation potentials. In this work, we propose a competition mechanism to address this fundamental challenge of representation collapse. By routing inputs only to experts with the highest neural response, we show that, under mild assumptions, competition enjoys the same convergence rate as the optimal estimator. We further propose CompeteSMoE, an effective and efficient algorithm to train large language models by deploying a simple router that predicts the competition outcomes. Consequently, CompeteSMoE enjoys strong performance gains from the competition routing policy while having low computation overheads. Our extensive empirical evaluations on two transformer architectures and a wide range of tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE compared to state-of-the-art SMoE strategies.

通过引入竞争机制来解决稀疏专家混合（SMoE）的表示坍塌问题，我们提出了CompeteSMoE算法，通过部署一个简单的路由器来预测竞争结果，从而在保持低计算开销的同时实现了强大的性能提升。我们在两个变压器架构和各种任务上进行了广泛的实证评估，表明CompeteSMoE相对于最先进的SMoE策略而言具有高效性、鲁棒性和可扩展性。

CompeteSMoE - 通过竞争有效训练稀疏专家混合模型