Large language models (LLMs) have recently achieved state-of-the-art
performance across various tasks, yet due to their large computational
requirements, they struggle with strict latency and power demands. Deep neural
network (DNN) quantization has traditionally addressed these limitations by
converting models to low-precision integer formats. Yet recently alternative
formats, such as Normal Float (NF4), have been shown to consistently increase
model accuracy, albeit at the cost of increased chip area. In this work, we
first conduct a large-scale analysis of LLM weights and activations across 30
networks to conclude most distributions follow a Student's t-distribution. We
then derive a new theoretically optimal format, Student Float (SF4), with
respect to this distribution, that improves over NF4 across modern LLMs, for
example increasing the average accuracy on LLaMA2-7B by 0.76% across tasks.
Using this format as a high-accuracy reference, we then propose augmenting E2M1
with two variants of supernormal support for higher model accuracy. Finally, we
explore the quality and performance frontier across 11 datatypes, including
non-traditional formats like Additive-Powers-of-Two (APoT), by evaluating their
model accuracy and hardware complexity. We discover a Pareto curve composed of
INT4, E2M1, and E2M1 with supernormal support, which offers a continuous
tradeoff between model accuracy and chip area. For example, E2M1 with
supernormal support increases the accuracy of Phi-2 by up to 2.19% with 1.22%
area overhead, enabling more LLM-based applications to be run at four bits.

使用大规模分析，我们得出结论：大多数分布遵循 t 分布。由此，我们推导出基于该分布的新型格式 -- Student Float（SF4），其相较于 Normal Float（NF4）可提高现代大语言模型的平均准确率。接着，我们提出通过增加两种超正常支持的方式来改进 E2M1 模型，进一步增强其准确率。最后，我们评估了 11 种数据类型的模型准确率和硬件复杂度，并发现 INT4、E2M1 和具有超正常支持的 E2M1 组成了一个帕累托曲线，可在模型准确率和芯片面积之间进行连续权衡，从而在 1.22％的面积开销下将 Phi-2 的准确率提高了高达 2.19％。

从学生中学习：应用 t - 分布来探索 LLM 的准确和高效格式

Learning from Students: Applying t-Distributions to Explore Accurate and  Efficient Formats for LLMs

Deep neural network (DNN) quantization converting floating-point (FP) data in
the network to integers (INT) is an effective way to shrink the model size for
memory saving and simplify the operations for compute acceleration. Recently,
researches on DNN quantization develop from inference to training, laying a
foundation for the online training on accelerators. However, existing schemes
leaving batch normalization (BN) untouched during training are mostly
incomplete quantization that still adopts high precision FP in some parts of
the data paths. Currently, there is no solution that can use only low bit-width
INT data during the whole training process of large-scale DNNs with acceptable
accuracy. In this work, through decomposing all the computation steps in DNNs
and fusing three special quantization functions to satisfy the different
precision requirements, we propose a unified complete quantization framework
termed as ``WAGEUBN'' to quantize DNNs involving all data paths including W
(Weights), A (Activation), G (Gradient), E (Error), U (Update), and BN.
Moreover, the Momentum optimizer is also quantized to realize a completely
quantized framework. Experiments on ResNet18/34/50 models demonstrate that
WAGEUBN can achieve competitive accuracy on the ImageNet dataset. For the first
time, the study of quantization in large-scale DNNs is advanced to the full
8-bit INT level. In this way, all the operations in the training and inference
can be bit-wise operations, pushing towards faster processing speed, decreased
memory cost, and higher energy efficiency. Our throughout quantization
framework has great potential for future efficient portable devices with online
learning ability.

本文提出了一种名为 WAGEUBN 的统一完整量化框架，可将神经网络的数据路径（包括所有数据类型 W，A，G，E，U 和 BN）中所有数据转换为低位整数以实现全面量化和在线训练，并在 ImageNet 数据集上获得了可比较的精度，展示了在大规模 DNNs 中 8 位 INT 级别的完整量化的功能性。