Audio classification is an active research area with a wide range of
applications. Over the past decade, convolutional neural networks (CNNs) have
been the de-facto standard building block for end-to-end audio classification
models. Recently, neural networks based solely on self-attention mechanisms
such as the Audio Spectrogram Transformer (AST) have been shown to outperform
CNNs. In this paper, we find an intriguing interaction between the two very
different models - CNN and AST models are good teachers for each other. When we
use either of them as the teacher and train the other model as the student via
knowledge distillation (KD), the performance of the student model noticeably
improves, and in many cases, is better than the teacher model. In our
experiments with this CNN/Transformer Cross-Model Knowledge Distillation (CMKD)
method we achieve new state-of-the-art performance on FSD50K, AudioSet, and
ESC-50.

本文研究了卷积神经网络和自注意力机制模型之间的交互作用，通过知识蒸馏将它们作为师生模型相互训练，在 FSD50K、AudioSet 和 ESC-50 等数据集上取得了新的最好成绩。