Large and sparse feed-forward networks (S-FFN) such as Mixture-of-Experts
(MoE) have demonstrated to be an efficient approach for scaling up Transformers
model size for pretraining large language models. By only activating part of
the FFN parameters conditioning on input, S-FFN improves generalization
performance while keeping training and inference costs (in FLOPs) fixed. In
this work, we analyzed the two major design choices of S-FFN: the memory block
(or expert) size and the memory block selection method under a general
conceptual framework of sparse neural memory. Using this unified framework, we
compare several S-FFN architectures for language modeling and provide insights
into their relative efficacy and efficiency. From our analysis results, we
found a simpler selection method -- Avg-K that selects blocks through their
mean aggregated hidden states, achieves lower perplexity in language modeling
pretraining compared to existing MoE architectures.

本文分析了 S-FFN 这种大而稀疏的前馈网络的两个主要设计选择：内存块（或专家）大小和内存块选择方法，并提供了它们相对有效性和效率的见解。在语言建模前训练中，我们发现一种更简单的选择方法 ——Avg-K，通过均值聚合隐藏状态选择块，可以比现有的 MoE 架构实现更低的困惑度。

大型语言模型预训练中稀疏前馈网络的统一视角

Towards A Unified View of Sparse Feed-Forward Network in Pretraining  Large Language Model

Neural memory enables fast adaptation to new tasks with just a few training
samples. Existing memory models store features only from the single last layer,
which does not generalize well in presence of a domain shift between training
and test distributions. Rather than relying on a flat memory, we propose a
hierarchical alternative that stores features at different semantic levels. We
introduce a hierarchical prototype model, where each level of the prototype
fetches corresponding information from the hierarchical memory. The model is
endowed with the ability to flexibly rely on features at different semantic
levels if the domain shift circumstances so demand. We meta-learn the model by
a newly derived hierarchical variational inference framework, where
hierarchical memory and prototypes are jointly optimized. To explore and
exploit the importance of different semantic levels, we further propose to
learn the weights associated with the prototype at each level in a data-driven
way, which enables the model to adaptively choose the most generalizable
features. We conduct thorough ablation studies to demonstrate the effectiveness
of each component in our model. The new state-of-the-art performance on
cross-domain and competitive performance on traditional few-shot classification
further substantiates the benefit of hierarchical variational memory.

本研究提出了一种基于分层原型模型和层次内存的神经记忆方法，该方法能够在领域转移的情况下，灵活地依赖不同语义层面的特征来适应新任务，而数据驱动的学习方法进一步可以使模型自适应地选择最具通用性的特征。该方法通过大量消融实验验证了性能的显著提升，尤其在跨领域和少样本分类方面表现出了显著优势。