This effort aims to reproduce the results of experiments and analyze the
robustness of the review framework for knowledge distillation introduced in the
CVPR '21 paper 'Distilling Knowledge via Knowledge Review' by Chen et al.
Previous works in knowledge distillation only studied connections paths between
the same levels of the student and the teacher, and cross-level connection
paths had not been considered. Chen et al. propose a new residual learning
framework to train a single student layer using multiple teacher layers. They
also design a novel fusion module to condense feature maps across levels and a
loss function to compare feature information stored across different levels to
improve performance. In this work, we consistently verify the improvements in
test accuracy across student models as reported in the original paper and study
the effectiveness of the novel modules introduced by conducting ablation
studies and new experiments.

本论文旨在复现 CVPR '21 论文《通过知识审查来提取知识》的实验结果并分析其知识蒸馏方法的健壮性。通过引入残差学习框架并设计新的多层教师神经网络的融合模块来训练单层学生神经网络，以及后引入的跨级连接方法，该方法取得了显著的性能提升。本文在通过消融实验和新实验验证了其有效性，并与原论文报告的测试精度进行了一致的验证。