Network compression is now a mature sub-field of neural network research:
over the last decade, significant progress has been made towards reducing the
size of models and speeding up inference, while maintaining the classification
accuracy. However, many works have observed that focusing on just the overall
accuracy can be misguided. E.g., it has been shown that mismatches between the
full and compressed models can be biased towards under-represented classes.
This raises the important research question, \emph{can we achieve network
compression while maintaining ``semantic equivalence'' with the original
network?} In this work, we study this question in the context of the ``long
tail'' phenomenon in computer vision datasets observed by Feldman, et al. They
argue that \emph{memorization} of certain inputs (appropriately defined) is
essential to achieving good generalization. As compression limits the capacity
of a network (and hence also its ability to memorize), we study the question:
are mismatches between the full and compressed models correlated with the
memorized training data? We present positive evidence in this direction for
image classification tasks, by considering different base architectures and
compression schemes.

本文研究在神经网络的网络压缩过程中如何保持原始网络的语义等价性，并通过对长尾现象的探讨，正向证明了全模型和压缩模型之间的差异与训练数据的记忆程度相关。