Scaling up the number of parameters of language models has proven to be an
effective approach to improve performance. For dense models, increasing model
size proportionally increases the model's computation footprint. In this work,
we seek to aggressively decouple learning capacity and FLOPs through
Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary
based routing functions and experts. Our proposed approach, dubbed Mixture of
Word Experts (MoWE), can be seen as a memory augmented model, where a large set
of word-specific experts play the role of a sparse memory. We demonstrate that
MoWE performs significantly better than the T5 family of models with similar
number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms
regular MoE models on knowledge intensive tasks and has similar performance to
more complex memory augmented approaches that often require to invoke custom
mechanisms to search the sparse memory.

该论文通过引入大型知识丰富词汇的路由功能和专家，以 Mixture-of-Experts (MoE) 风格模型，积极地解耦学习容量与 FLOPs，提出了一种名为 Mixture of Word Experts (MoWE) 的方法，它可以被视为一种记忆增强模型，其中一组特定于单词的专家起到了稀疏记忆的作用，研究表明，在各种 NLP 任务中，MoWE 的性能明显优于相似 FLOPs 数量的 T5 模型系列，此外，MoWE 在知识密集型任务上胜过常规的 MoE 模型，并且具有与更复杂的记忆增强方法相似的性能，而后者通常需要调用自定义机制来搜索稀疏记忆。