In this paper, we explore FP8 low-bit data formats for efficient training of
large language models (LLMs). Our key insight is that most variables, such as
gradients and optimizer states, in LLM training can employ low-precision data
formats without compromising model accuracy and requiring no changes to
hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision
framework for training LLMs. This framework offers three levels of FP8
utilization to streamline mixed-precision and distributed parallel training for
LLMs. It gradually incorporates 8-bit gradients, optimizer states, and
distributed learning in an incremental manner. Experiment results show that,
during the training of GPT-175B model on H100 GPU platform, our FP8
mixed-precision training framework not only achieved a remarkable 42% reduction
in real memory usage but also ran 64% faster than the widely adopted BF16
framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer
Engine by 17%. This largely reduces the training costs for large foundation
models. Furthermore, our FP8 mixed-precision training methodology is generic.
It can be seamlessly applied to other tasks such as LLM instruction tuning and
reinforcement learning with human feedback, offering savings in fine-tuning
expenses. Our FP8 low-precision training framework is open-sourced at
{this https URL}{aka.ms/MS.AMP}.

我们探索了 FP8 低位数据格式以实现大规模语言模型（LLM）的高效训练。我们提出了一种新的 FP8 自动混合精度框架，可以在递增的方式下渐进地应用 8 位梯度、优化器状态和分布式学习。实验证明，在 H100 GPU 平台上使用我们的 FP8 混合精度训练框架，与广泛采用的 BF16 框架相比，GPT-175B 模型的训练不仅可以实现 42% 的实际内存使用降低，而且运行速度比 Nvidia Transformer Engine 快 64%，超过其 17%。此外，我们的 FP8 混合精度训练方法具有通用性，可以无缝应用于 LLM 指导调优、带有人类反馈的强化学习等任务，节省了微调开销。