The Mixture-of-Experts (MoE) layer, a sparsely-activated model controlled by
a router, has achieved great success in deep learning. However, the
understanding of such architecture remains elusive. In this paper, we formally
study how the MoE layer improves the performance of neural network learning and
why the mixture model will not collapse into a single model. Our empirical
results suggest that the cluster structure of the underlying problem and the
non-linearity of the expert are pivotal to the success of MoE. To further
understand this, we consider a challenging classification problem with
intrinsic cluster structures, which is hard to learn using a single expert. Yet
with the MoE layer, by choosing the experts as two-layer nonlinear
convolutional neural networks (CNNs), we show that the problem can be learned
successfully. Furthermore, our theory shows that the router can learn the
cluster-center features, which helps divide the input complex problem into
simpler linear classification sub-problems that individual experts can conquer.
To our knowledge, this is the first result towards formally understanding the
mechanism of the MoE layer for deep learning.

本文研究了 Mixture-of-Experts（MoE）层如何在深度学习中提高神经网络的性能以及为什么混合模型不会崩溃。通过研究困难的分类问题，我们证明了 MoE 模型的有效性，理论上表明路由器可以学习聚类中心特征，帮助将复杂的问题分解为更简单的线性分类子问题，而专家网络可以解决这些子问题，该文是最早正式理解深度学习中 MoE 层机制的结果之一。