Knowledge Distillation (KD) aims to learn a compact student network using
knowledge from a large pre-trained teacher network, where both networks are
trained on data from the same distribution. However, in practical applications,
the student network may be required to perform in a new scenario (i.e., the
target domain), which usually exhibits significant differences from the known
scenario of the teacher network (i.e., the source domain). The traditional
domain adaptation techniques can be integrated with KD in a two-stage process
to bridge the domain gap, but the ultimate reliability of two-stage approaches
tends to be limited due to the high computational consumption and the
additional errors accumulated from both stages. To solve this problem, we
propose a new one-stage method dubbed ``Direct Distillation between Different
Domains" (4Ds). We first design a learnable adapter based on the Fourier
transform to separate the domain-invariant knowledge from the domain-specific
knowledge. Then, we build a fusion-activation mechanism to transfer the
valuable domain-invariant knowledge to the student network, while
simultaneously encouraging the adapter within the teacher network to learn the
domain-specific knowledge of the target data. As a result, the teacher network
can effectively transfer categorical knowledge that aligns with the target
domain of the student network. Intensive experiments on various benchmark
datasets demonstrate that our proposed 4Ds method successfully produces
reliable student networks and outperforms state-of-the-art approaches.

基于知识蒸馏，提出了一种新的单阶段方法 “不同领域之间的直接蒸馏”(4Ds)，通过可学习的适配器和融合 - 激活机制，实现了从教师网络到学生网络的跨领域知识转移，取得了可靠的学生网络性能并超过了现有方法。

不同领域之间的直接蒸馏

Direct Distillation between Different Domains

Knowledge distillation (KD) has proved to be an effective approach for deep
neural network compression, which learns a compact network (student) by
transferring the knowledge from a pre-trained, over-parameterized network
(teacher). In traditional KD, the transferred knowledge is usually obtained by
feeding training samples to the teacher network to obtain the class
probabilities. However, the original training dataset is not always available
due to storage costs or privacy issues. In this study, we propose a novel
data-free KD approach by modeling the intermediate feature space of the teacher
with a multivariate normal distribution and leveraging the soft targeted labels
generated by the distribution to synthesize pseudo samples as the transfer set.
Several student networks trained with these synthesized transfer sets present
competitive performance compared to the networks trained with the original
training set and other data-free KD approaches.

本研究提出了一个基于多元正态分布的数据自由知识蒸馏方法，该方法从过度参数化神经网络中模拟中间特征空间并利用软目标标签生成伪样本以作为转换集。使用这些合成的转移集训练多个学生网络，表现出与使用原始训练集和其他无数据知识蒸馏方法训练的网络相当的竞争性能。