Pruning before training enables the deployment of neural networks on smart
devices. By retaining weights conducive to generalization, pruned networks can
be accommodated on resource-constrained smart devices. It is commonly held that
the distance on weight norms between the initialized and the fully-trained
networks correlates with generalization performance. However, as we have
uncovered, inconsistency between this metric and generalization during training
processes, which poses an obstacle to determine the pruned structures on smart
devices in advance. In this paper, we introduce the concept of the learning
gap, emphasizing its accurate correlation with generalization. Experiments show
that the learning gap, in the form of feature maps from the penultimate layer
of networks, aligns with variations of generalization performance. We propose a
novel learning framework, LNPT, which enables mature networks on the cloud to
provide online guidance for network pruning and learning on smart devices with
unlabeled data. Our results demonstrate the superiority of this approach over
supervised training.

在本文中，我们介绍了学习差距的概念，并强调其与泛化能力的准确相关性。实验表明，学习差距以网络倒数第二层的特征图的形式与泛化性能的变化相一致。我们提出了一种新的学习框架，LNPT，使得云端的成熟网络能够对没有标签的智能设备上的网络修剪和学习提供在线指导。我们的结果证明了这种方法优于监督训练。