The introduction of large-scale audio datasets, such as AudioSet, paved the
way for Transformers to conquer the audio domain and replace CNNs as the
state-of-the-art neural network architecture for many tasks. Audio Spectrogram
Transformers are excellent at exploiting large datasets, creating powerful
pre-trained models that surpass CNNs when fine-tuned on downstream tasks.
However, current popular Audio Spectrogram Transformers are demanding in terms
of computational complexity compared to CNNs. Recently, we have shown that, by
employing Transformer-to-CNN Knowledge Distillation, efficient CNNs can catch
up with and even outperform Transformers on large datasets. In this work, we
extend this line of research and increase the capacity of efficient CNNs by
introducing dynamic CNN blocks, constructed of dynamic non-linearities, dynamic
convolutions and attention mechanisms. We show that these dynamic CNNs
outperform traditional efficient CNNs, in terms of the performance-complexity
trade-off and parameter efficiency, at the task of audio tagging on the
large-scale AudioSet. Our experiments further indicate that the introduced
dynamic CNNs achieve better performance on downstream tasks and scale up well,
attaining Transformer performance and even outperforming them on AudioSet and
several downstream tasks.

在大规模音频数据集中，通过 Transformer 取代 CNN 作为神经网络体系结构的最先进模型，本文通过 Transformer 与 CNN 的知识蒸馏，提出了一种高效的 CNN 模型，通过引入动态非线性、动态卷积和注意机制构建了动态 CNN 块，实验证明这种动态 CNN 模型在音频标签任务中的性能复杂度权衡和参数效率方面超越了传统的高效 CNN 模型，并且在下游任务中表现出更好的性能，达到了 Transformer 的性能甚至在 AudioSet 和多个下游任务上超越了 Transformer。