Transformer-based models, such as BERT, have been widely applied in a wide
range of natural language processing tasks. However, one inevitable side effect
is that they require massive memory storage and inference cost when deployed in
production. Quantization is one of the popularized ways to alleviate the cost.
However, the previous 8-bit quantization strategy based on INT8 data format
either suffers from the degradation of accuracy in a Post-Training Quantization
(PTQ) fashion or requires an expensive Quantization-Aware Training (QAT)
process. Recently, a new numeric format FP8 (i.e. floating-point of 8-bits) has
been proposed and supported in commercial AI computing platforms such as H100.
In this paper, we empirically validate the effectiveness of FP8 as a way to do
Post-Training Quantization without significant loss of accuracy, with a simple
calibration and format conversion process. We adopt the FP8 standard proposed
by~\citet{nvidia_release} in our extensive experiments of BERT variants on GLUE
and SQuAD v1.1 datasets, and show that PTQ with FP8 can significantly improve
the accuracy upon that with INT8, to the extent of the full-precision model.

本文通过对 GLUE 和 SQuAD v1.1 数据集上的 BERT 变种进行大量实验，验证了 FP8 作为一种进行后训练量化的有效方式，能够在不显著损失精度的情况下提高准确性。

FP8-BERT：Transformer 的后训练量化

FP8-BERT: Post-Training Quantization for Transformer

When quantizing neural networks for efficient inference, low-bit integers are
the go-to format for efficiency. However, low-bit floating point numbers have
an extra degree of freedom, assigning some bits to work on an exponential scale
instead. This paper in-depth investigates this benefit of the floating point
format for neural network inference. We detail the choices that can be made for
the FP8 format, including the important choice of the number of bits for the
mantissa and exponent, and show analytically in which settings these choices
give better performance. Then we show how these findings translate to real
networks, provide an efficient implementation for FP8 simulation, and a new
algorithm that enables the learning of both the scale parameters and the number
of exponent bits in the FP8 format. Our chief conclusion is that when doing
post-training quantization for a wide range of networks, the FP8 format is
better than INT8 in terms of accuracy, and the choice of the number of exponent
bits is driven by the severity of outliers in the network. We also conduct
experiments with quantization-aware training where the difference in formats
disappears as the network is trained to reduce the effect of outliers.

本研究深入探究了采用浮点数格式（FP8）进行神经网络量化的性能优势，分析了尾数和指数位数选择对性能的影响，并通过大量实验证明，相对于 INT8 格式，FP8 格式具有更高的精度和更优的性能。