Adaptive inference is a promising technique to improve the computational
efficiency of deep models at test time. In contrast to static models which use
the same computation graph for all instances, adaptive networks can dynamically
adjust their structure conditioned on each input. While existing research on
adaptive inference mainly focuses on designing more advanced architectures,
this paper investigates how to train such networks more effectively.
Specifically, we consider a typical adaptive deep network with multiple
intermediate classifiers. We present three techniques to improve its training
efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the
conflict of learning of different classifiers; 2) an Inline Subnetwork
Collaboration approach and a One-for-all Knowledge Distillation algorithm to
enhance the collaboration among classifiers. On multiple datasets (CIFAR-10,
CIFAR-100 and ImageNet), we show that the proposed approach consistently leads
to further improved efficiency on top of state-of-the-art adaptive deep
networks.

本文研究了如何更有效地训练具有多个中间分类器的自适应深度网络，提出了梯度均衡算法，内联子网络协作方法和一对所有知识蒸馏算法等三种方法，以改进分类器协作并进一步提高自适应性深度网络的效率。在 CIFAR-10，CIFAR-100 和 ImageNet 等多个数据集上展示了该方法相对于现有技术的改进效果。