All-MLP architectures have attracted increasing interest as an alternative to
attention-based models. In NLP, recent work like gMLP shows that all-MLPs can
match Transformers in language modeling, but still lag behind in downstream
tasks. In this work, we analyze the limitations of MLPs in expressiveness, and
propose sparsely activated MLPs with mixture-of-experts (MoEs) in both feature
and input (token) dimensions. Such sparse all-MLPs significantly increase model
capacity and expressiveness while keeping the compute constant. We address
critical challenges in incorporating conditional computation with two routing
strategies. The proposed sparse all-MLP improves language modeling perplexity
and obtains up to 2$\times$ improvement in training efficiency compared to both
Transformer-based MoEs (GShard, Switch Transformer, Base Layers and HASH
Layers) as well as dense Transformers and all-MLPs. Finally, we evaluate its
zero-shot in-context learning performance on six downstream tasks, and find
that it surpasses Transformer-based MoEs and dense Transformers.

该论文提出稀疏激活多层感知机结合混合专家模型在语言建模任务上显著提高模型容量和表达能力，同时保持运算速度稳定，相较于 Transformer-based MoEs、密集型 Transformer 和全连接 MLP 在训练效率和语言建模复杂度方面获得了 2 倍提升。并在六项下游任务中取得了优于 Transformer-based MoEs 和密集型 Transformer 的效果。