Reviewing plays an important role when learning knowledge. The knowledge
acquisition at a certain time point may be strongly inspired with the help of
previous experience. Thus the knowledge growing procedure should show strong
relationship along the temporal dimension. In our research, we find that during
the network training, the evolution of feature map follows temporal sequence
property. A proper temporal supervision may further improve the network
training performance. Inspired by this observation, we design a novel knowledge
distillation method. Specifically, we extract the spatiotemporal features in
the different training phases of student by convolutional Long Short-term
memory network (Conv-LSTM). Then, we train the student net through a dynamic
target, rather than static teacher network features. This process realizes the
refinement of old knowledge in student network, and utilizes them to assist
current learning. Extensive experiments verify the effectiveness and advantages
of our method over existing knowledge distillation methods, including various
network architectures, different tasks (image classification and object
detection) .

研究发现，在网络训练期间，特征映射的演化遵循时间顺序属性，使用卷积长短期记忆网络（Conv-LSTM）提取学生的不同训练阶段的时空特征进行动态目标训练，可以实现在学生网络中对旧知识的精炼并利用它们来帮助当前的学习，相关实验证明其方法相对于现有的知识蒸馏方法在各种网络架构和不同任务上的有效性和优势。