This paper examines the quantization methods used in large-scale data
analysis models and their hyperparameter choices. The recent surge in data
analysis scale has significantly increased computational resource requirements.
To address this, quantizing model weights has become a prevalent practice in
data analysis applications such as deep learning. Quantization is particularly
vital for deploying large models on devices with limited computational
resources. However, the selection of quantization hyperparameters, like the
number of bits and value range for weight quantization, remains an
underexplored area. In this study, we employ the typical case analysis from
statistical physics, specifically the replica method, to explore the impact of
hyperparameters on the quantization of simple learning models. Our analysis
yields three key findings: (i) an unstable hyperparameter phase, known as
replica symmetry breaking, occurs with a small number of bits and a large
quantization width; (ii) there is an optimal quantization width that minimizes
error; and (iii) quantization delays the onset of overparameterization, helping
to mitigate overfitting as indicated by the double descent phenomenon. We also
discover that non-uniform quantization can enhance stability. Additionally, we
develop an approximate message-passing algorithm to validate our theoretical
results.

本文研究了大规模数据分析模型中使用的量化方法及其超参数选择。通过采用统计物理学中的典型案例分析，特别是副本方法，我们探索了超参数对简单学习模型的量化产生的影响，并得到了三个关键发现：（i）在位数较少和量化宽度较大的情况下会出现不稳定的超参数相位，即副本对称性破缺；（ii）存在一个最佳的量化宽度能够最小化误差；以及（iii）量化有助于延迟过参数化的发生，减轻了过拟合问题，这一点可以从双下降现象看出。此外，我们还发现非均匀量化可以增强稳定性，并开发了一个近似传递信息算法来验证我们的理论结果。