Training large deep learning models at scale is very challenging. This paper
proposes Chimera, a novel pipeline parallelism scheme which combines
bidirectional pipelines for efficiently training large-scale models. Chimera is
a synchronous approach and therefore no loss of accuracy, which is more
convergence-friendly than asynchronous approaches. Compared with the latest
synchronous pipeline approach, Chimera reduces the number of bubbles by up to
50%; benefiting from the sophisticated scheduling of bidirectional pipelines,
Chimera has a more balanced activation memory consumption. Evaluations are
conducted on Transformer based language models. For a GPT-2 model with 1.3
billion parameters running on 2,048 GPU nodes of the Piz Daint supercomputer,
Chimera improves the training throughput by 1.16x-2.34x over the
state-of-the-art synchronous and asynchronous pipeline approaches.

本研究提出了一种名为 Chimera 的新型管道并行方案，采用双向管道以有效训练大型深度学习模型，该方法是一种同步方法，相对于异步方法更加收敛友好，相对于最新的同步并行方法，Chimera 可以将冒泡数量减少多达 50％ 。在基于 Transformer 的语言模型上进行评估，对于在 Piz Daint 超级计算机的 2,048 个 GPU 节点上运行的 13 亿参数的 GPT-2 模型，Chimera 将训练吞吐量提高了 1.16 倍 - 2.34 倍。