As the training of giant dense models hits the boundary on the availability
and capability of the hardware resources today, Mixture-of-Experts (MoE) models
become one of the most promising model architectures due to their significant
training cost reduction compared to a quality-equivalent dense model. Its
training cost saving is demonstrated from encoder-decoder models (prior works)
to a 5x saving for auto-aggressive language models (this work along with
parallel explorations). However, due to the much larger model size and unique
architecture, how to provide fast MoE model inference remains challenging and
unsolved, limiting its practical usage. To tackle this, we present
DeepSpeed-MoE, an end-to-end MoE training and inference solution as part of the
DeepSpeed library, including novel MoE architecture designs and model
compression techniques that reduce MoE model size by up to 3.7x, and a highly
optimized inference system that provides 7.3x better latency and cost compared
to existing MoE inference solutions. DeepSpeed-MoE offers an unprecedented
scale and efficiency to serve massive MoE models with up to 4.5x faster and 9x
cheaper inference compared to quality-equivalent dense models. We hope our
innovations and systems help open a promising path to new directions in the
large model landscape, a shift from dense to sparse MoE models, where training
and deploying higher-quality models with fewer resources becomes more widely
possible.

本文介绍了一种通过 Mixture-of-Experts 模型架构实现的深度学习模型训练及推断方案，名为 DeepSpeed-MoE。该方案通过新颖的模型压缩技术和高度优化的推断系统，在减小模型尺寸、提高能效和降低硬件资源要求等方面都有着非常显著的表现。希望通过 Sparse MoE Models 的训练和部署，开辟更多高质量模型的新方向，减少硬件资源的需求，加速模型的落地应用。