Mixture-of-Experts (MoE) has gained increasing popularity as a promising
framework for scaling up large language models (LLMs). However, training MoE
from scratch in a large-scale setting still suffers from data-hungry and
instability problems. Motivated by this limit, we investigate building MoE
models from existing dense large language models. Specifically, based on the
well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert
Construction, which partitions the parameters of original Feed-Forward Networks
(FFNs) into multiple experts; (2) Continual Pre-training, which further trains
the transformed MoE model and additional gate networks. In this paper, we
comprehensively explore different methods for expert construction and various
data sampling strategies for continual pre-training. After these stages, our
LLaMA-MoE models could maintain language abilities and route the input tokens
to specific experts with part of the parameters activated. Empirically, by
training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense
models that contain similar activation parameters. The source codes and models
are available at this https URL .

通过从现有的大型语言模型构建混合专家（MoE）模型，我们提出了一种从头开始训练在大规模环境下仍面临数据需求和不稳定性问题的 MoE 的方法。我们综合探索了不同的专家构建方法和各种数据采样策略，最终生成的 LLaMA-MoE 模型能够保持语言能力并将输入标记路由到具体的专家。经验证明，在训练 200B 标记后，LLaMA-MoE-3.5B 模型在含有相似激活参数的密集模型上表现出显著优势。