Data-free knowledge distillation (DFKD) aims to distill pretrained knowledge
to a student model with the help of a generator without using original data. In
such data-free scenarios, achieving stable performance of DFKD is essential due
to the unavailability of validation data. Unfortunately, this paper has
discovered that existing DFKD methods are quite sensitive to different teacher
models, occasionally showing catastrophic failures of distillation, even when
using well-trained teacher models. Our observation is that the generator in
DFKD is not always guaranteed to produce precise yet diverse samples using the
existing representative strategy of minimizing both class-prior and adversarial
losses. Through our empirical study, we focus on the fact that class-prior not
only decreases the diversity of generated samples, but also cannot completely
address the problem of generating unexpectedly low-quality samples depending on
teacher models. In this paper, we propose the teacher-agnostic data-free
knowledge distillation (TA-DFKD) method, with the goal of more robust and
stable performance regardless of teacher models. Our basic idea is to assign
the teacher model a lenient expert role for evaluating samples, rather than a
strict supervisor that enforces its class-prior on the generator. Specifically,
we design a sample selection approach that takes only clean samples verified by
the teacher model without imposing restrictions on the power of generating
diverse samples. Through extensive experiments, we show that our method
successfully achieves both robustness and training stability across various
teacher models, while outperforming the existing DFKD methods.

无数据知识蒸馏方法 (TA-DFKD）通过为生成器分配宽松的专家角色，而非严格的监督者角色，以更健壮和稳定的性能，实现了在各种教师模型下的蒸馏，并超越了现有的无数据知识蒸馏方法。