Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, many of these strategies have complex interactions regarding the final training efficiency. Prior work tackling this problem did not have access to the latest set of optimizations, such as FlashAttention or sequence parallelism. In this work, we conduct a comprehensive ablation study of possible training configurations for large language models. We distill this large study into several key recommendations for the most efficient training. For instance, we find that using a micro-batch size of 1 usually enables the most efficient training layouts. Larger micro-batch sizes necessitate activation checkpointing or higher degrees of model parallelism and also lead to larger pipeline bubbles. Our most efficient configurations enable us to achieve state-of-the-art training efficiency results over a range of model sizes, most notably a Model FLOPs utilization of 70.5% when training a 13B model.

通过对大型语言模型的多种训练配置进行全面的消融研究，我们总结出几个关键建议，以实现最高效的训练，例如，我们发现使用微批量大小为1通常能实现最高效的训练布局。较大的微批量大小需要激活检查点或更高程度的模型并行性，并且还会导致更大的管道延迟。我们最高效的配置使得在一系列模型规模上实现了最先进的训练效率结果，尤其是在训练13B模型时，模型FLOPs利用率达到了70.5%。

大规模分布式模型训练的高效并行化布局