Optimizing a deep neural network is a fundamental task in computer vision,
yet direct training methods often suffer from over-fitting. Teacher-student
optimization aims at providing complementary cues from a model trained
previously, but these approaches are often considerably slow due to the
pipeline of training a few generations in sequence, i.e., time complexity is
increased by several times.
This paper presents snapshot distillation (SD), the first framework which
enables teacher-student optimization in one generation. The idea of SD is very
simple: instead of borrowing supervision signals from previous generations, we
extract such information from earlier epochs in the same generation, meanwhile
make sure that the difference between teacher and student is sufficiently large
so as to prevent under-fitting. To achieve this goal, we implement SD in a
cyclic learning rate policy, in which the last snapshot of each cycle is used
as the teacher for all iterations in the next cycle, and the teacher signal is
smoothed to provide richer information. In standard image classification
benchmarks such as CIFAR100 and ILSVRC2012, SD achieves consistent accuracy
gain without heavy computational overheads. We also verify that models
pre-trained with SD transfers well to object detection and semantic
segmentation in the PascalVOC dataset.

本研究提出了快照蒸馏 (SD) 框架，它将深度神经网络的教师 - 学生优化方法应用于一个生成周期内，通过循环学习率策略以及利用同一周期内早期时期的信息提取，避免了时间复杂度的增大，同时在 CIFAR100 和 ILSVRC2012 等基准测试中获得了准确性的一致提升。