Data-free knowledge distillation (DFKD) is a promising approach for
addressing issues related to model compression, security privacy, and
transmission restrictions. Although the existing methods exploiting DFKD have
achieved inspiring achievements in coarse-grained classification, in practical
applications involving fine-grained classification tasks that require more
detailed distinctions between similar categories, sub-optimal results are
obtained. To address this issue, we propose an approach called DFKD-FGVC that
extends DFKD to fine-grained visual categorization~(FGVC) tasks. Our approach
utilizes an adversarial distillation framework with attention generator, mixed
high-order attention distillation, and semantic feature contrast learning.
Specifically, we introduce a spatial-wise attention mechanism to the generator
to synthesize fine-grained images with more details of discriminative parts. We
also utilize the mixed high-order attention mechanism to capture complex
interactions among parts and the subtle differences among discriminative
features of the fine-grained categories, paying attention to both local
features and semantic context relationships. Moreover, we leverage the teacher
and student models of the distillation framework to contrast high-level
semantic feature maps in the hyperspace, comparing variances of different
categories. We evaluate our approach on three widely-used FGVC benchmarks
(Aircraft, Cars196, and CUB200) and demonstrate its superior performance.

我们提出了一种名为 DFKD-FGVC 的方法，它扩展了 DFKD 到细粒度视觉分类任务，使用了对抗蒸馏框架、注意力生成器、混合高阶注意力蒸馏和语义特征对比学习，以实现更好的性能表现。