Despite the impressive performance of deep neural networks (DNNs), their
computational complexity and storage space consumption have led to the concept
of network compression. While DNN compression techniques such as pruning and
low-rank decomposition have been extensively studied, there has been
insufficient attention paid to their theoretical explanation. In this paper, we
propose a novel theoretical framework that leverages a probabilistic latent
space of DNN weights and explains the optimal network sparsity by using the
information-theoretic divergence measures. We introduce new analogous projected
patterns (AP2) and analogous-in-probability projected patterns (AP3) notions
for DNNs and prove that there exists a relationship between AP3/AP2 property of
layers in the network and its performance. Further, we provide a theoretical
analysis that explains the training process of the compressed network. The
theoretical results are empirically validated through experiments conducted on
standard pre-trained benchmarks, including AlexNet, ResNet50, and VGG16, using
CIFAR10 and CIFAR100 datasets. Through our experiments, we highlight the
relationship of AP3 and AP2 properties with fine-tuning pruned DNNs and
sparsity levels.

通过使用信息论分歧度量，本文提出了一个基于深度神经网络 (DNNs) 权重的概率潜在空间的新的理论框架，解释了网络稀疏性与性能之间的关系，并通过实验证明了这个理论的结果。