In real applications, different computation-resource devices need
different-depth networks (e.g., ResNet-18/34/50) with high-accuracy. Usually,
existing methods either design multiple networks and train them independently,
or construct depth-level/width-level dynamic neural networks which is hard to
prove the accuracy of each sub-net. In this article, we propose an elegant
Depth-Level Dynamic Neural Network (DDNN) integrated different-depth sub-nets
of similar architectures. To improve the generalization of sub-nets, we design
the Embedded-Knowledge-Distillation (EKD) training mechanism for the DDNN to
implement knowledge transfer from the teacher (full-net) to multiple students
(sub-nets). Specifically, the Kullback-Leibler (KL) divergence is introduced to
constrain the posterior class probability consistency between full-net and
sub-nets, and self-attention distillation on the same resolution feature of
different depth is addressed to drive more abundant feature representations of
sub-nets. Thus, we can obtain multiple high-accuracy sub-nets simultaneously in
a DDNN via the online knowledge distillation in each training iteration without
extra computation cost. Extensive experiments on CIFAR-10/100, and ImageNet
datasets demonstrate that sub-nets in DDNN with EKD training achieve better
performance than individually training networks while preserving the original
performance of full-nets.

本研究提出了一种嵌入知识蒸馏机制的深度级联动态神经网络 (DDNN)，通过 Kullback-Leibler 散度和自注意力蒸馏在在线知识蒸馏中获得多个高精度子网络，以提高子网络的泛化能力，从而在 CIFAR-10/100 和 ImageNet 数据集上实现了 better performance than individually training networks while preserving the original performance of full-nets。