Data-Free Knowledge Distillation (DFKD) is a novel task that aims to train
high-performance student models using only the teacher network without original
training data. Despite encouraging results, existing DFKD methods rely heavily
on generation modules with high computational costs. Meanwhile, they ignore the
fact that the generated and original data exist domain shifts due to the lack
of supervision information. Moreover, knowledge is transferred through each
example, ignoring the implicit relationship among multiple examples. To this
end, we propose a novel Open-world Data Sampling Distillation (ODSD) method
without a redundant generation process. First, we try to sample open-world data
close to the original data's distribution by an adaptive sampling module. Then,
we introduce a low-noise representation to alleviate the domain shifts and
build a structured relationship of multiple data examples to exploit data
knowledge. Extensive experiments on CIFAR-10, CIFAR-100, NYUv2, and ImageNet
show that our ODSD method achieves state-of-the-art performance. Especially, we
improve 1.50\%-9.59\% accuracy on the ImageNet dataset compared with the
existing results.

提出了一种名为数据无关知识蒸馏（DFKD）的新任务，旨在使用仅带有原始训练数据的教师网络训练高性能学生模型。通过自适应采样模块对接近原始数据分布的开放世界数据进行采样，引入低噪声表示以减轻领域漂移并建立多个数据示例的结构化关系来开发数据知识。在 CIFAR-10、CIFAR-100、NYUv2 和 ImageNet 等广泛实验中，我们的 ODSD 方法实现了最先进的性能，在 ImageNet 数据集上与现有结果相比，准确率提高了 1.50％-9.59％。