Transformer models trained on long sequences often achieve higher accuracy
than short sequences. Unfortunately, conventional transformers struggle with
long sequence training due to the overwhelming computation and memory
requirements. Existing methods for long sequence training offer limited speedup
and memory reduction, and may compromise accuracy. This paper presents a novel
and efficient distributed training method, the Long Short-Sequence Transformer
(LSS Transformer), for training transformer with long sequences. It distributes
a long sequence into segments among GPUs, with each GPU computing a partial
self-attention for its segment. Then, it uses a fused communication and a novel
double gradient averaging technique to avoid the need to aggregate partial
self-attention and minimize communication overhead. We evaluated the
performance between LSS Transformer and the state-of-the-art Nvidia sequence
parallelism on a Wikipedia enwik8 dataset. Results show that our proposed
method lead to 5.6x faster and 10.2x more memory-efficient implementation
compared to state-of-the-art sequence parallelism on 144 Nvidia V100 GPUs.
Moreover, our algorithm scales to an extreme sequence length of 50,112 at 3,456
GPUs, achieving 161% super-linear parallel efficiency and a throughput of 32
petaflops.

这篇论文提出了一种新颖高效的分布式训练方法，使用长短序列变压器（LSS Transformer）来训练长序列的变压器，将长序列分布到多个 GPU 上进行计算，并通过融合通信和双梯度平均技术来提高训练效率和减少通信开销。与最先进的序列并行方法相比，在 Wikipedia enwik8 数据集上，我们的方法在 144 个 Nvidia V100 GPU 上实现了 5.6 倍的加速和 10.2 倍的内存效率，且在 3,456 个 GPU 上可扩展到长度达到 50,112 的极限序列，实现了 161% 的超线性并行效率和 32 petaflops 的吞吐量。