Scaling up deep neural network capacity has been known as an effective
approach to improving model quality for several different machine learning
tasks. In many cases, increasing model capacity beyond the memory limit of a
single accelerator has required developing special algorithms or
infrastructure. These solutions are often architecture-specific and do not
transfer to other tasks. To address the need for efficient and task-independent
model parallelism, we introduce GPipe, a pipeline parallelism library that
allows scaling any network that can be expressed as a sequence of layers. By
pipelining different sub-sequences of layers on separate accelerators, GPipe
provides the flexibility of scaling a variety of different networks to gigantic
sizes efficiently. Moreover, GPipe utilizes a novel batch-splitting pipelining
algorithm, resulting in almost linear speedup when a model is partitioned
across multiple accelerators. We demonstrate the advantages of GPipe by
training large-scale neural networks on two different tasks with distinct
network architectures: (i) Image Classification: We train a
557-million-parameter AmoebaNet model and attain a top-1 accuracy of 84.4% on
ImageNet-2012, (ii) Multilingual Neural Machine Translation: We train a single
6-billion-parameter, 128-layer Transformer model on a corpus spanning over 100
languages and achieve better quality than all bilingual models.

GPipe 是一种管道并行库，它提供了一种高效进行模型并行化的方式，可以将不同子层序列分别放在不同的加速器上，因此可以有效地将各种不同的网络扩展到巨大的规模，并且能够实现快速的线性加速，通过对两个不同任务的训练，证明了 GPipe 的优势。