We show that model compression can improve the population risk of a
pre-trained model, by studying the tradeoff between the decrease in the
generalization error and the increase in the empirical risk with model
compression. We first prove that model compression reduces an
information-theoretic bound on the generalization error; this allows for an
interpretation of model compression as a regularization technique to avoid
overfitting. We then characterize the increase in empirical risk with model
compression using rate distortion theory. These results imply that the
population risk could be improved by model compression if the decrease in
generalization error exceeds the increase in empirical risk. We show through a
linear regression example that such a decrease in population risk due to model
compression is indeed possible. Our theoretical results further suggest that
the Hessian-weighted $K$-means clustering compression approach can be improved
by regularizing the distance between the clustering centers. We provide
experiments with neural networks to support our theoretical assertions.

通过压缩模型，平衡泛化误差的减小和经验风险的增加，从而证明模型压缩可以提高预训练模型的总体风险，同时推荐使用 Hessian-weighted K-means 聚类压缩方法进行正则化，通过神经网络的实验进一步验证了理论结论。