The sparsely activated mixture of experts (MoE) model presents a promising
alternative to traditional densely activated (dense) models, enhancing both
quality and computational efficiency. However, training MoE models from scratch
demands extensive data and computational resources. Moreover, public
repositories like timm mainly provide pre-trained dense checkpoints, lacking
similar resources for MoE models, hindering their adoption. To bridge this gap,
we introduce MoE Jetpack, an effective method for fine-tuning dense checkpoints
into MoE models. MoE Jetpack incorporates two key techniques: (1) checkpoint
recycling, which repurposes dense checkpoints as initial weights for MoE
models, thereby accelerating convergence, enhancing accuracy, and alleviating
the computational burden of pre-training; (2) hyperspherical adaptive MoE
(SpheroMoE) layer, which optimizes the MoE architecture for better integration
of dense checkpoints, enhancing fine-tuning performance. Our experiments on
vision tasks demonstrate that MoE Jetpack significantly improves convergence
speed and accuracy when fine-tuning dense checkpoints into MoE models. Our code
will be publicly available at this https URL

我们介绍了 MoE Jetpack，这是一种将密集检查点优化为 MoE 模型的有效方法。MoE Jetpack 包括两个关键技术：(1) 检查点回收，将密集检查点重新用于 MoE 模型的初始权重，以加速收敛、提高准确性并减轻预训练的计算负担；(2) 球形自适应 MoE (SpheroMoE) 层，为更好地融合密集检查点而优化 MoE 架构，提高精细调整性能。我们的实验证明 MoE Jetpack 在视觉任务中将密集检查点优化为 MoE 模型时显著提高了收敛速度和准确性。