Knowledge Distillation (KD) aims to learn a compact student network using
knowledge from a large pre-trained teacher network, where both networks are
trained on data from the same distribution. However, in practical applications,
the student network may be required to perform in a new scenario (i.e., the
target domain), which usually exhibits significant differences from the known
scenario of the teacher network (i.e., the source domain). The traditional
domain adaptation techniques can be integrated with KD in a two-stage process
to bridge the domain gap, but the ultimate reliability of two-stage approaches
tends to be limited due to the high computational consumption and the
additional errors accumulated from both stages. To solve this problem, we
propose a new one-stage method dubbed ``Direct Distillation between Different
Domains" (4Ds). We first design a learnable adapter based on the Fourier
transform to separate the domain-invariant knowledge from the domain-specific
knowledge. Then, we build a fusion-activation mechanism to transfer the
valuable domain-invariant knowledge to the student network, while
simultaneously encouraging the adapter within the teacher network to learn the
domain-specific knowledge of the target data. As a result, the teacher network
can effectively transfer categorical knowledge that aligns with the target
domain of the student network. Intensive experiments on various benchmark
datasets demonstrate that our proposed 4Ds method successfully produces
reliable student networks and outperforms state-of-the-art approaches.

基于知识蒸馏，提出了一种新的单阶段方法 “不同领域之间的直接蒸馏”(4Ds)，通过可学习的适配器和融合 - 激活机制，实现了从教师网络到学生网络的跨领域知识转移，取得了可靠的学生网络性能并超过了现有方法。