Knowledge distillation is effective to train small and generalisable network
models for meeting the low-memory and fast running requirements. Existing
offline distillation methods rely on a strong pre-trained teacher, which
enables favourable knowledge discovery and transfer but requires a complex
two-phase training procedure. Online counterparts address this limitation at
the price of lacking a highcapacity teacher. In this work, we present an
On-the-fly Native Ensemble (ONE) strategy for one-stage online distillation.
Specifically, ONE trains only a single multi-branch network while
simultaneously establishing a strong teacher on-the- fly to enhance the
learning of target network. Extensive evaluations show that ONE improves the
generalisation performance a variety of deep neural networks more significantly
than alternative methods on four image classification dataset: CIFAR10,
CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency
advantages.

本文研究使用在线知识蒸馏方法，提出了一种 On-the-fly Native Ensemble (ONE) 策略，通过构建强大的 teacher 来提高目标网络的学习，该方法在多种深度神经网络上性能明显优于其他方法，并具有计算效率优势。