The Mixture of Experts (MoE) for language models has been proven effective in
augmenting the capacity of models by dynamically routing each input token to a
specific subset of experts for processing. Despite the success, most existing
methods face a challenge for balance between sparsity and the availability of
expert knowledge: enhancing performance through increased use of expert
knowledge often results in diminishing sparsity during expert selection. To
mitigate this contradiction, we propose HyperMoE, a novel MoE framework built
upon Hypernetworks. This framework integrates the computational processes of
MoE with the concept of knowledge transferring in multi-task learning. Specific
modules generated based on the information of unselected experts serve as
supplementary information, which allows the knowledge of experts not selected
to be used while maintaining selection sparsity. Our comprehensive empirical
evaluations across multiple datasets and backbones establish that HyperMoE
significantly outperforms existing MoE methods under identical conditions
concerning the number of experts.

HyperMoE 是一种基于 Hypernetworks 的新型 Mixture of Experts (MoE) 框架，通过利用未选择的专家生成的特定模块作为补充信息，实现在保持选择稀疏性的同时利用未选择的专家的知识，从而在相同条件下显著优于现有 MoE 方法。