Large language models (LLMs) have recently achieved state-of-the-art
performance across various tasks, yet due to their large computational
requirements, they struggle with strict latency and power demands. Deep neural
network (DNN) quantization has traditionally addressed these limitations by
converting models to low-precision integer formats. Yet recently alternative
formats, such as Normal Float (NF4), have been shown to consistently increase
model accuracy, albeit at the cost of increased chip area. In this work, we
first conduct a large-scale analysis of LLM weights and activations across 30
networks to conclude most distributions follow a Student's t-distribution. We
then derive a new theoretically optimal format, Student Float (SF4), with
respect to this distribution, that improves over NF4 across modern LLMs, for
example increasing the average accuracy on LLaMA2-7B by 0.76% across tasks.
Using this format as a high-accuracy reference, we then propose augmenting E2M1
with two variants of supernormal support for higher model accuracy. Finally, we
explore the quality and performance frontier across 11 datatypes, including
non-traditional formats like Additive-Powers-of-Two (APoT), by evaluating their
model accuracy and hardware complexity. We discover a Pareto curve composed of
INT4, E2M1, and E2M1 with supernormal support, which offers a continuous
tradeoff between model accuracy and chip area. For example, E2M1 with
supernormal support increases the accuracy of Phi-2 by up to 2.19% with 1.22%
area overhead, enabling more LLM-based applications to be run at four bits.

使用大规模分析，我们得出结论：大多数分布遵循 t 分布。由此，我们推导出基于该分布的新型格式 -- Student Float（SF4），其相较于 Normal Float（NF4）可提高现代大语言模型的平均准确率。接着，我们提出通过增加两种超正常支持的方式来改进 E2M1 模型，进一步增强其准确率。最后，我们评估了 11 种数据类型的模型准确率和硬件复杂度，并发现 INT4、E2M1 和具有超正常支持的 E2M1 组成了一个帕累托曲线，可在模型准确率和芯片面积之间进行连续权衡，从而在 1.22％的面积开销下将 Phi-2 的准确率提高了高达 2.19％。