Knowledge Distillation is a technique which aims to utilize dark knowledge to
compress and transfer information from a vast, well-trained neural network
(teacher model) to a smaller, less capable neural network (student model) with
improved inference efficiency. This approach of distilling knowledge has gained
popularity as a result of the prohibitively complicated nature of such
cumbersome models for deployment on edge computing devices. Generally, the
teacher models used to teach smaller student models are cumbersome in nature
and expensive to train. To eliminate the necessity for a cumbersome teacher
model completely, we propose a simple yet effective knowledge distillation
framework that we termed Dynamic Rectification Knowledge Distillation (DR-KD).
Our method transforms the student into its own teacher, and if the self-teacher
makes wrong predictions while distilling information, the error is rectified
prior to the knowledge being distilled. Specifically, the teacher targets are
dynamically tweaked by the agency of ground-truth while distilling the
knowledge gained from traditional training. Our proposed DR-KD performs
remarkably well in the absence of a sophisticated cumbersome teacher model and
achieves comparable performance to existing state-of-the-art teacher-free
knowledge distillation frameworks when implemented by a low-cost dynamic
mannered teacher. Our approach is all-encompassing and can be utilized for any
deep neural network training that requires categorization or object
recognition. DR-KD enhances the test accuracy on Tiny ImageNet by 2.65% over
prominent baseline models, which is significantly better than any other
knowledge distillation approach while requiring no additional training costs.

提出了一种名为 DR-KD 的简单而有效的知识蒸馏框架，通过在传统训练中动态调整目标来使学生变成自身的教师，并在蒸馏信息时进行错误更正，从而消除了笨重的教师模型的必要性，并在需要分类或对象识别的任何深度神经网络训练中均可使用，能够在不增加额外训练成本的情况下较其他知识蒸馏方法显著提高 Tiny ImageNet 测试准确性 2.65％。