The increasing size of deep neural networks necessitates effective model compression to improve computational efficiency and reduce their memory footprint. Sparsity and quantization are two prominent compression methods that have individually demonstrated significant reduction in computational and memory footprints while preserving model accuracy. While effective, the interplay between these two methods remains an open question. In this paper, we investigate the interaction between these two methods and assess whether their combination impacts final model accuracy. We mathematically prove that applying sparsity before quantization is the optimal sequence for these operations, minimizing error in computation. Our empirical studies across a wide range of models, including OPT and Llama model families (125M-8B) and ViT corroborate these theoretical findings. In addition, through rigorous analysis, we demonstrate that sparsity and quantization are not orthogonal; their interaction can significantly harm model accuracy, with quantization error playing a dominant role in this degradation. Our findings extend to the efficient deployment of large models in resource-limited compute platforms and reduce serving cost, offering insights into best practices for applying these compression methods to maximize efficacy without compromising accuracy.

深度神经网络的模型压缩是提高计算效率和减小内存占用的必要手段。本文通过数学证明和实证研究，发现在计算操作中先应用稀疏化再进行量化是最优的操作次序，以最小化计算误差。同时，稀疏化和量化的相互作用会对模型精度造成重要的影响，其中量化误差在这一降低中起到主导作用。研究结果对于在资源有限的计算平台上高效部署大型模型和降低计算成本非常有价值，为应用这些压缩方法以最大化效果而不损害准确性提供了指导。

稀疏性与量化的有效相互作用：从理论到实践