Recent work has shown that feed-forward networks (FFNs) in pre-trained
Transformers are a key component, storing various linguistic and factual
knowledge. However, the computational patterns of FFNs are still unclear. In
this work, we study the computational patterns of FFNs and observe that most
inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is
similar to the sparsity of the human brain, which drives research on functional
partitions of the human brain. To verify whether functional partitions also
emerge in FFNs, we propose to convert a model into its MoE version with the
same parameters, namely MoEfication. Specifically, MoEfication consists of two
phases: (1) splitting the parameters of FFNs into multiple functional
partitions as experts, and (2) building expert routers to decide which experts
will be used for each input. Experimental results show that MoEfication can
conditionally use 10% to 30% of FFN parameters while maintaining over 95%
original performance for different models on various downstream tasks. Besides,
MoEfication brings two advantages: (1) it significantly reduces the FLOPS of
inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a
fine-grained perspective to study the inner mechanism of FFNs. The source code
of this paper can be obtained from this https URL

本研究探讨了预训练模型 Transformers 中前馈网络（FFN）的计算模式，提出了一种将模型 MoEfication 为多个功能分区的方法，并建立了专家路由器以决定每个输入使用哪个专家。实验结果表明，这种方法可以在保持原始性能的同时使用 10％至 30％的 FFN 参数，同时提高了推断 FLOPS 的效率和提供了一个细粒度的视角来研究 FFN 的内部机制。