Recent mainstream masked distillation methods function by reconstructing
selectively masked areas of a student network from the feature map of its
teacher counterpart. In these methods, the masked regions need to be properly
selected, such that reconstructed features encode sufficient discrimination and
representation capability like the teacher feature. However, previous masked
distillation methods only focus on spatial masking, making the resulting masked
areas biased towards spatial importance without encoding informative channel
clues. In this study, we devise a Dual Masked Knowledge Distillation (DMKD)
framework which can capture both spatially important and channel-wise
informative clues for comprehensive masked feature reconstruction. More
specifically, we employ dual attention mechanism for guiding the respective
masking branches, leading to reconstructed feature encoding dual significance.
Furthermore, fusing the reconstructed features is achieved by self-adjustable
weighting strategy for effective feature distillation. Our experiments on
object detection task demonstrate that the student networks achieve performance
gains of 4.1% and 4.3% with the help of our method when RetinaNet and Cascade
Mask R-CNN are respectively used as the teacher networks, while outperforming
the other state-of-the-art distillation methods.

利用双重掩蔽知识蒸馏（DMKD）框架来捕捉空间重要性和通道相关信息以实现全面的掩蔽特征重构，通过自适应加权策略融合重构特征进行有效的特征蒸馏，在目标检测任务中与其他最先进的蒸馏方法相比，学生网络分别在 RetinaNet 和级联 Mask R-CNN 作为教师网络时取得了 4.1% 和 4.3% 的性能提升。