Model quantification uses low bit-width values to represent the weight
matrices of models, which is a promising approach to reduce both storage and
computational overheads of deploying highly anticipated LLMs. However, existing
quantization methods suffer severe performance degradation when the bit-width
is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to
quantize models. This paper boldly quantizes the weight matrices of LLMs to
1-bit, paving the way for the extremely low bit-width deployment of LLMs. For
this target, we introduce a 1-bit quantization-aware training (QAT) framework
named OneBit, including a novel 1-bit parameter representation method to better
quantize LLMs as well as an effective parameter initialization method based on
matrix decomposition to improve the convergence speed of the QAT framework.
Sufficient experimental results indicate that OneBit achieves good performance
(at least 83% of the non-quantized performance) with robust training processes
when only using 1-bit weight matrices.

该研究使用 1 位量化来减少高度期望的低精度模型的存储和计算开销，并通过引入一种 1 位量化感知训练框架 OneBit 以及基于矩阵分解的参数初始化方法来实现良好的性能（至少达到非量化性能的 83%）。

OneBit: 极低位大型语言模型

OneBit: Towards Extremely Low-bit Large Language Models

Running Stochastic Gradient Descent (SGD) in a decentralized fashion has
shown promising results. In this paper we propose Moniqua, a technique that
allows decentralized SGD to use quantized communication. We prove in theory
that Moniqua communicates a provably bounded number of bits per iteration,
while converging at the same asymptotic rate as the original algorithm does
with full-precision communication. Moniqua improves upon prior works in that it
(1) requires zero additional memory, (2) works with 1-bit quantization, and (3)
is applicable to a variety of decentralized algorithms. We demonstrate
empirically that Moniqua converges faster with respect to wall clock time than
other quantized decentralized algorithms. We also show that Moniqua is robust
to very low bit-budgets, allowing 1-bit-per-parameter communication without
compromising validation accuracy when training ResNet20 and ResNet110 on
CIFAR10.

本文提出了 Moniqua 技术，它允许分散式的 SGD 使用量化通信，在不降低精度的前提下，每次迭代只需要传输一定数量的信息。此技术不需要额外内存，可适用于多种去中心化算法，并且在实验中已经证明了相对于其他量化去中心化算法更快的收敛速度，且在极低的比特预算下仍然表现出较高的准确率。