In this technical report, we introduce the training methodologies implemented
in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)
large language model (LLM) with 146 billion parameters and 16 experts. It is
initialized from the pre-existing dense checkpoints of our Skywork-13B model.
We explore the comparative effectiveness of upcycling versus training from
scratch initializations. Our findings suggest that the choice between these two
approaches should consider both the performance of the existing dense
checkpoints and the MoE training budget. We highlight two innovative
techniques: gating logit normalization, which improves expert diversification,
and adaptive auxiliary loss coefficients, allowing for layer-specific
adjustment of auxiliary loss coefficients. Our experimental results validate
the effectiveness of these methods. Leveraging these techniques and insights,
we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile
corpus. The evaluation results demonstrate that our model delivers strong
performance across a wide range of benchmarks.

在本研究中，我们介绍了 Skywork-MoE 的训练方法，它是一个具有 1460 亿参数和 16 个专家的高性能混合专家（MoE）大型语言模型（LLM），它是从我们的 Skywork-13B 模型的预先存在的密集检查点初始化的。我们对再利用与从头训练初始化之间的比较效果进行了探索研究，发现这两种方法的选择应该考虑到现有密集检查点的性能和 MoE 训练预算。我们突出了两个创新技术：门控逻辑归一化，可以改善专家多样化，以及自适应辅助损失系数，可以对每层进行调整。我们的实验结果验证了这些方法的有效性。利用这些技术和见解，我们对我们的 SkyPile 语料库的一个压缩子集进行了再利用训练，评估结果证明我们的模型在广泛的基准测试中表现出强大的性能。