Network quantization is one of network compression techniques to reduce the
redundancy of deep neural networks. It reduces the number of distinct network
parameter values by quantization in order to save the storage for them. In this
paper, we design network quantization schemes that minimize the performance
loss due to quantization given a compression ratio constraint. We analyze the
quantitative relation of quantization errors to the neural network loss
function and identify that the Hessian-weighted distortion measure is locally
the right objective function for the optimization of network quantization. As a
result, Hessian-weighted k-means clustering is proposed for clustering network
parameters to quantize. When optimal variable-length binary codes, e.g.,
Huffman codes, are employed for further compression, we derive that the network
quantization problem can be related to the entropy-constrained scalar
quantization (ECSQ) problem in information theory and consequently propose two
solutions of ECSQ for network quantization, i.e., uniform quantization and an
iterative solution similar to Lloyd's algorithm. Finally, using the simple
uniform quantization followed by Huffman coding, we show from our experiments
that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet,
32-layer ResNet and AlexNet, respectively.

本研究提出了一种基于 Hessian-weighted k-means clustering 和 ECSQ 的网络量化方案，以在达到压缩比约束的前提下最小化网络量化的性能损失，并进行了 LeNet，32 层 ResNet 和 AlexNet 的压缩实验。