Scaling the size of language models usually leads to remarkable advancements
in NLP tasks. But it often comes with a price of growing computational cost.
Although a sparse Mixture of Experts (MoE) can reduce the cost by activating a
small subset of parameters (e.g., one expert) for each input, its computation
escalates significantly if increasing the number of activated experts, limiting
its practical utility. Can we retain the advantages of adding more experts
without substantially increasing the computational costs? In this paper, we
first demonstrate the superiority of selecting multiple experts and then
propose a computation-efficient approach called \textbf{\texttt{Merging Experts
into One}} (MEO), which reduces the computation cost to that of a single
expert. Extensive experiments show that MEO significantly improves
computational efficiency, e.g., FLOPS drops from 72.0G of vanilla MoE to 28.6G
(MEO). Moreover, we propose a token-level attention block that further enhances
the efficiency and performance of token-level MEO, e.g., 83.3\% (MEO) vs.
82.6\% (vanilla MoE) average score on the GLUE benchmark. Our code will be
released upon acceptance. Code will be released at:
https://github.com/Shwai-He/MEO.

本研究主要通过提出一种计算较为高效的方法 Merging Experts into One（MEO）来降低计算成本，并通过实验证明其优越性以及如何进一步提高效率和性能。