As the development of neural networks, more and more deep neural networks are
adopted in various tasks, such as image classification. However, as the huge
computational overhead, these networks could not be applied on mobile devices
or other low latency scenes. To address this dilemma, multi-classifier
convolutional network is proposed to allow faster inference via early
classifiers with the corresponding classifiers. These networks utilize
sophisticated designing to increase the early classifier accuracy. However,
naively training the multi-classifier network could hurt the performance
(accuracy) of deep neural networks as early classifiers throughout interfere
with the feature generation process.
In this paper, we propose a general training framework named
multi-self-distillation learning (MSD), which mining knowledge of different
classifiers within the same network and increase every classifier accuracy. Our
approach can be applied not only to multi-classifier networks, but also modern
CNNs (e.g., ResNet Series) augmented with additional side branch classifiers.
We use sampling-based branch augmentation technique to transform a
single-classifier network into a multi-classifier network. This reduces the gap
of capacity between different classifiers, and improves the effectiveness of
applying MSD. Our experiments show that MSD improves the accuracy of various
networks: enhancing the accuracy of every classifier significantly for existing
multi-classifier network (MSDNet), improving vanilla single-classifier networks
with internal classifiers with high accuracy, while also improving the final
accuracy.

本文提出了一个名为多自我蒸馏学习 (MSD) 的通用训练框架，通过在同一网络中挖掘不同分类器的知识来提高每个分类器的准确性。我们使用基于采样的分支增强技术将单分类器网络转换为多分类器网络，实验结果表明 MSD 可以显着提高整个网络的准确性。

多自我蒸馏学习：基于深度神经网络内多分类器的学习

MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep  Neural Networks

Deep neural networks are state of the art methods for many learning tasks due
to their ability to extract increasingly better features at each network layer.
However, the improved performance of additional layers in a deep network comes
at the cost of added latency and energy usage in feedforward inference. As
networks continue to get deeper and larger, these costs become more prohibitive
for real-time and energy-sensitive applications. To address this issue, we
present BranchyNet, a novel deep network architecture that is augmented with
additional side branch classifiers. The architecture allows prediction results
for a large portion of test samples to exit the network early via these
branches when samples can already be inferred with high confidence. BranchyNet
exploits the observation that features learned at an early layer of a network
may often be sufficient for the classification of many data points. For more
difficult samples, which are expected less frequently, BranchyNet will use
further or all network layers to provide the best likelihood of correct
prediction. We study the BranchyNet architecture using several well-known
networks (LeNet, AlexNet, ResNet) and datasets (MNIST, CIFAR10) and show that
it can both improve accuracy and significantly reduce the inference time of the
network.

本文介绍一种新颖的深度网络架构 ——BranchyNet，该架构通过引入额外的支路分类器，允许预测结果提前退出网络，实现较高置信度下的快速推理，以此同时提高网络的分类准确度，并有效地降低网络的推理时间。