Large and sparse feed-forward networks (S-FFN) such as Mixture-of-Experts
(MoE) have demonstrated to be an efficient approach for scaling up Transformers
model size for pretraining large language models. By only activating part of
the FFN parameters conditioning on input, S-FFN improves generalization
performance while keeping training and inference costs (in FLOPs) fixed. In
this work, we analyzed the two major design choices of S-FFN: the memory block
(or expert) size and the memory block selection method under a general
conceptual framework of sparse neural memory. Using this unified framework, we
compare several S-FFN architectures for language modeling and provide insights
into their relative efficacy and efficiency. From our analysis results, we
found a simpler selection method -- Avg-K that selects blocks through their
mean aggregated hidden states, achieves lower perplexity in language modeling
pretraining compared to existing MoE architectures.

本文分析了 S-FFN 这种大而稀疏的前馈网络的两个主要设计选择：内存块（或专家）大小和内存块选择方法，并提供了它们相对有效性和效率的见解。在语言建模前训练中，我们发现一种更简单的选择方法 ——Avg-K，通过均值聚合隐藏状态选择块，可以比现有的 MoE 架构实现更低的困惑度。