The Mixture-of-Expert (MoE) technique plays a crucial role in expanding the
size of DNN model parameters. However, it faces the challenge of extended
all-to-all communication latency during the training process. Existing methods
attempt to mitigate this issue by overlapping all-to-all with expert
computation. Yet, these methods frequently fall short of achieving sufficient
overlap, consequently restricting the potential for performance enhancements.
In our study, we extend the scope of this challenge by considering overlap at
the broader training graph level. During the forward pass, we enable non-MoE
computations to overlap with all-to-all through careful partitioning and
pipelining. In the backward pass, we achieve overlap with all-to-all by
scheduling gradient weight computations. We implement these techniques in
Lancet, a system using compiler-based optimization to automatically enhance MoE
model training. Our extensive evaluation reveals that Lancet significantly
reduces the time devoted to non-overlapping communication, by as much as 77%.
Moreover, it achieves a notable end-to-end speedup of up to 1.3 times when
compared to the state-of-the-art solutions.

我们的研究扩展了 Mixture-of-Expert (MoE) 技术在训练过程中的重叠性挑战，并通过特定的分区和流水线技术实现了非 MoE 计算与 all-to-all 通信的重叠。我们的 Lancet 系统在自动增强 MoE 模型训练时，显著减少了非重叠性通信所需的时间，最高达到 77%。此外，与同类解决方案相比，速度提升达到了显著的 1.3 倍。