Data-Free Knowledge Distillation (DFKD) has recently attracted growing
attention in the academic community, especially with major breakthroughs in
computer vision. Despite promising results, the technique has not been well
applied to audio and signal processing. Due to the variable duration of audio
signals, it has its own unique way of modeling. In this work, we propose
feature-rich audio model inversion (FRAMI), a data-free knowledge distillation
framework for general sound classification tasks. It first generates
high-quality and feature-rich Mel-spectrograms through a feature-invariant
contrastive loss. Then, the hidden states before and after the statistics
pooling layer are reused when knowledge distillation is performed on these
feature-rich samples. Experimental results on the Urbansound8k, ESC-50, and
audioMNIST datasets demonstrate that FRAMI can generate feature-rich samples.
Meanwhile, the accuracy of the student model is further improved by reusing the
hidden state and significantly outperforms the baseline method.

本文提出了一种名为 FRAMI 的无数据知识蒸馏框架来进行通用音频分类任务，该方法利用 Mel-spectrograms 生成高质量且具有特征丰富性的音频样本，其在音频分类上的实验表明，通过重用隐藏状态可以显着提升学生模型的准确性。