Convolutional neural networks have been widely deployed in various
application scenarios. In order to extend the applications' boundaries to some
accuracy-crucial domains, researchers have been investigating approaches to
boost accuracy through either deeper or wider network structures, which brings
with them the exponential increment of the computational and storage cost,
delaying the responding time. In this paper, we propose a general training
framework named self distillation, which notably enhances the performance
(accuracy) of convolutional neural networks through shrinking the size of the
network rather than aggrandizing it. Different from traditional knowledge
distillation - a knowledge transformation methodology among networks, which
forces student neural networks to approximate the softmax layer outputs of
pre-trained teacher neural networks, the proposed self distillation framework
distills knowledge within network itself. The networks are firstly divided into
several sections. Then the knowledge in the deeper portion of the networks is
squeezed into the shallow ones. Experiments further prove the generalization of
the proposed self distillation framework: enhancement of accuracy at average
level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as
maximum. In addition, it can also provide flexibility of depth-wise scalable
inference on resource-limited edge devices.Our codes will be released on github
soon.

提出了一种名为 “自蒸馏” 的卷积神经网络训练框架，通过将网络大小缩小而不是扩大来显著提高卷积神经网络的性能（准确性）。它与传统的知识蒸馏不同，后者是将预训练的教师神经网络的输出作为 softmax 层输出的近似值强制学生神经网络去逼近。该框架将知识内化到网络本身，对深度方面的可伸缩推理提供了灵活性，能够在资源有限的边缘设备上运行。

自我蒸馏：通过蒸馏提高卷积神经网络性能

Be Your Own Teacher: Improve the Performance of Convolutional Neural  Networks via Self Distillation

Deep convolutional networks are well-known for their high computational and
memory demands. Given limited resources, how does one design a network that
balances its size, training time, and prediction accuracy? A surprisingly
effective approach to trade accuracy for size and speed is to simply reduce the
number of channels in each convolutional layer by a fixed fraction and retrain
the network. In many cases this leads to significantly smaller networks with
only minimal changes to accuracy. In this paper, we take a step further by
empirically examining a strategy for deactivating connections between filters
in convolutional layers in a way that allows us to harvest savings both in
run-time and memory for many network architectures. More specifically, we
generalize 2D convolution to use a channel-wise sparse connection structure and
show that this leads to significantly better results than the baseline approach
for large networks including VGG and Inception V3.

通过实验证明了通过在卷积层之间停用连接的方法将采用通道稀疏连接结构的 2D 卷积技术与许多网络体系结构一起使用，可以在保持准确性的同时实现明显的运行时间和内存储存器的节省。