Sparse Mixtures of Experts (SMoE) scales model capacity without significant
increases in training and inference costs, but exhibits the following two
issues: (1) Low expert activation, where only a small subset of experts are
activated for optimization. (2) Lacking fine-grained analytical capabilities
for multiple semantic concepts within individual tokens. We propose Multi-Head
Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each
token into multiple sub-tokens. These sub-tokens are then assigned to and
processed by a diverse set of experts in parallel, and seamlessly reintegrated
into the original token form. The multi-head mechanism enables the model to
collectively attend to information from various representation spaces within
different experts, while significantly enhances expert activation, thus deepens
context understanding and alleviate overfitting. Moreover, our MH-MoE is
straightforward to implement and decouples from other SMoE optimization
methods, making it easy to integrate with other SMoE models for enhanced
performance. Extensive experimental results across three tasks: English-focused
language modeling, Multi-lingual language modeling and Masked multi-modality
modeling tasks, demonstrate the effectiveness of MH-MoE.

多头专家混合模型 (MH-MoE) 使用多头机制将每个令牌拆分为多个子令牌，并并行分配给和处理多样的专家集，然后无缝地重新集成到原始令牌形式，从而增强了训练中的专家激活，深化了上下文理解，减轻过拟合，同时易于实施和与其他 SMoE 模型集成，对英语和多语言模型以及多模态建模任务进行了大量实验，证明了 MH-MoE 的有效性。