Large model training has been using recomputation to alleviate the memory
pressure and pipelining to exploit the parallelism of data, tensor, and
devices. The existing recomputation approaches may incur up to 40% overhead
when training real-world models, e.g., the GPT model with 22B parameters. This
is because they are executed on demand in the critical training path. In this
paper, we design a new recomputation framework, Lynx, to reduce the overhead by
overlapping the recomputation with communication occurring in training
pipelines. It consists of an optimal scheduling algorithm (OPT) and a
heuristic-based scheduling algorithm (HEU). OPT achieves a global optimum but
suffers from a long search time. HEU was designed based on our observation that
there are identical structures in large DNN models so that we can apply the
same scheduling policy to all identical structures. HEU achieves a local
optimum but reduces the search time by 99% compared to OPT. Our comprehensive
evaluation using GPT models with 1.3B-20B parameters shows that both OPT and
HEU outperform the state-of-the-art recomputation approaches (e.g., Megatron-LM
and Checkmake) by 1.02-1.53x. HEU achieves a similar performance as OPT with a
search time of 0.16s on average.

设计了一个新的重计算框架 Lynx，通过将重计算与训练管道中发生的通信重叠来减少开销，其中包括优化调度算法（OPT）和基于启发式的调度算法（HEU），OPT 实现了全局最优但搜索时间较长，HEU 通过应用相同调度策略于所有相同结构达到局部最优并将搜索时间减少了 99%，使用 1.3B-20B 参数的 GPT 模型的全面评估结果显示，OPT 和 HEU 在性能上都优于现有的重计算方法（例如 Megatron-LM 和 Checkmake）1.02-1.53 倍，HEU 的平均搜索时间为 0.16 秒，与 OPT 具有类似的性能。

通过重叠激活重计算优化大型模型训练

Optimizing Large Model Training through Overlapped Activation  Recomputation

In recent years, Mixture-of-Experts (MoE) has emerged as a promising
technique for deep learning that can scale the model capacity to trillion-plus
parameters while reducing the computing cost via sparse computation. While MoE
opens a new frontier of exceedingly large models, its implementation over
thousands of GPUs has been limited due to mismatch between the dynamic nature
of MoE and static parallelism/pipelining of the system. We present Tutel, a
highly scalable stack design and implementation for MoE with dynamically
adaptive parallelism and pipelining. Tutel delivers adaptive parallelism
switching and adaptive pipelining at runtime, which achieves up to 1.74x and
2.00x single MoE layer speedup, respectively. We also propose a novel
two-dimensional hierarchical algorithm for MoE communication speedup that
outperforms the previous state-of-the-art up to 20.7x over 2,048 GPUs.
Aggregating all techniques, Tutel finally delivers 4.96x and 5.75x speedup of a
single MoE layer on 16 GPUs and 2,048 GPUs, respectively, over Fairseq: Meta's
Facebook AI Research Sequence-to-Sequence Toolkit (Tutel is now partially
adopted by Fairseq). Tutel source code is available in public:
this https URL . Our evaluation shows that Tutel
efficiently and effectively runs a real-world MoE-based model named SwinV2-MoE,
built upon Swin Transformer V2, a state-of-the-art computer vision
architecture. On efficiency, Tutel accelerates SwinV2-MoE, achieving up to
1.55x and 2.11x speedup in training and inference over Fairseq, respectively.
On effectiveness, the SwinV2-MoE model achieves superior accuracy in both
pre-training and down-stream computer vision tasks such as COCO object
detection than the counterpart dense model, indicating the readiness of Tutel
for end-to-end real-world model training and inference. SwinV2-MoE is open
sourced in this https URL .

Tutel is a highly scalable stack design for Mixture-of-Experts (MoE) with dynamically adaptive parallelism and pipelining that achieves up to a 5.75x speedup of a single MoE layer on 2,048 GPUs over Fairseq, and delivers efficiency and effectiveness in running a real-world MoE-based model named SwinV2-MoE.