Knowledge distillation (KD) is a widely adopted and effective method for
compressing models in object detection tasks. Particularly, feature-based
distillation methods have shown remarkable performance. Existing approaches
often ignore the uncertainty in the teacher model's knowledge, which stems from
data noise and imperfect training. This limits the student model's ability to
learn latent knowledge, as it may overly rely on the teacher's imperfect
guidance. In this paper, we propose a novel feature-based distillation paradigm
with knowledge uncertainty for object detection, termed "Uncertainty
Estimation-Discriminative Knowledge Extraction-Knowledge Transfer (UET)", which
can seamlessly integrate with existing distillation methods. By leveraging the
Monte Carlo dropout technique, we introduce knowledge uncertainty into the
training process of the student model, facilitating deeper exploration of
latent knowledge. Our method performs effectively during the KD process without
requiring intricate structures or extensive computational resources. Extensive
experiments validate the effectiveness of our proposed approach across various
distillation strategies, detectors, and backbone architectures. Specifically,
following our proposed paradigm, the existing FGD method achieves
state-of-the-art (SoTA) performance, with ResNet50-based GFL achieving 44.1%
mAP on the COCO dataset, surpassing the baselines by 3.9%.

提出一种基于特征的知识不确定性蒸馏范式，能够与现有的蒸馏方法无缝集成，通过蒙特卡洛 dropout 技术引入知识不确定性，提高学生模型对潜在知识的探索能力，并在目标检测任务中获得有效性验证。

教学中的不确定性：释放目标检测知识蒸馏的潜力

Teaching with Uncertainty: Unleashing the Potential of Knowledge  Distillation in Object Detection

Knowledge distillation is commonly employed to compress neural networks,
reducing the inference costs and memory footprint. In the scenario of
homogenous architecture, feature-based methods have been widely validated for
their effectiveness. However, in scenarios where the teacher and student models
are of heterogeneous architectures, the inherent differences in feature
representation significantly degrade the performance of these methods. Recent
studies have highlighted that low-frequency components constitute the majority
of image features. Motivated by this, we propose a Low-Frequency
Components-based Contrastive Knowledge Distillation (LFCC) framework that
significantly enhances the performance of feature-based distillation between
heterogeneous architectures. Specifically, we designe a set of multi-scale
low-pass filters to extract the low-frequency components of intermediate
features from both the teacher and student models, aligning them in a compact
space to overcome architectural disparities. Moreover, leveraging the intrinsic
pairing characteristic of the teacher-student framework, we design an
innovative sample-level contrastive learning framework that adeptly
restructures the constraints of within-sample feature similarity and
between-sample feature divergence into a contrastive learning task. This
strategy enables the student model to capitalize on intra-sample feature
congruence while simultaneously enhancing the discrimination of features among
disparate samples. Consequently, our LFCC framework accurately captures the
commonalities in feature representation across heterogeneous architectures.
Extensive evaluations and empirical analyses across three architectures (CNNs,
Transformers, and MLPs) demonstrate that LFCC achieves superior performance on
the challenging benchmarks of ImageNet-1K and CIFAR-100. All codes will be
publicly available.

基于低频部分的对比知识蒸馏框架能够更好地在异构架构下提取特征表示的共性。通过使用多尺度低通滤波器提取教师和学生模型中间特征的低频部分，并通过对比学习任务优化学生模型的特征区分度，该框架在 ImageNet-1K 和 CIFAR-100 数据集上展现出卓越的性能。

在紧凑空间中对齐：异构架构之间的对比知识蒸馏

Aligning in a Compact Space: Contrastive Knowledge Distillation between  Heterogeneous Architectures

Knowledge Distillation (KD) for Convolutional Neural Network (CNN) is
extensively studied as a way to boost the performance of a small model.
Recently, Vision Transformer (ViT) has achieved great success on many computer
vision tasks and KD for ViT is also desired. However, besides the output
logit-based KD, other feature-based KD methods for CNNs cannot be directly
applied to ViT due to the huge structure gap. In this paper, we explore the way
of feature-based distillation for ViT. Based on the nature of feature maps in
ViT, we design a series of controlled experiments and derive three practical
guidelines for ViT's feature distillation. Some of our findings are even
opposite to the practices in the CNN era. Based on the three guidelines, we
propose our feature-based method ViTKD which brings consistent and considerable
improvement to the student. On ImageNet-1k, we boost DeiT-Tiny from 74.42% to
76.06%, DeiT-Small from 80.55% to 81.95%, and DeiT-Base from 81.76% to 83.46%.
Moreover, ViTKD and the logit-based KD method are complementary and can be
applied together directly. This combination can further improve the performance
of the student. Specifically, the student DeiT-Tiny, Small, and Base achieve
77.78%, 83.59%, and 85.41%, respectively. The code is available at
this https URL

本文研究 Vision Transformer 的特征蒸馏方法，并提出 ViTKD 可以使基于 ImageNet 数据集的学生模型的准确率分别提升 1.64％、1.4％和 1.7％。