In this paper, we propose a simple yet powerful improvement over the recent
Self-Supervised Audio Spectrogram Transformer (SSAST) model for speech and
audio classification. Specifically, we leverage the insight that the SSAST uses
a very high masking ratio (75%) during pretraining, meaning that the vast
majority of self-attention compute is performed on mask tokens. We address this
by integrating the encoder-decoder architecture from Masked Autoencoders are
Scalable Vision Learners (MAE) into the SSAST, where a deep encoder operates on
only unmasked input, and a shallow decoder operates on encoder outputs and mask
tokens. We find that MAE-like pretraining can provide a 3x speedup and 2x
memory usage reduction over the vanilla SSAST using current audio pretraining
strategies with ordinary model and input sizes. When fine-tuning on downstream
tasks, which only uses the encoder, we find that our approach outperforms the
SSAST on a variety of downstream tasks. We further conduct comprehensive
evaluations into different strategies of pretraining and explore differences in
MAE-style pretraining between the visual and audio domains.

本文提出了一种针对自我监督语音及音频分类中 Self-Supervised Audio Spectrogram Transformer (SSAST) 模型的简单、且功能强大的改进方法。具体而言，我们将来自原模型中使用的高掩模比率（75%）的问题进行改进，并将 Masked Autoencoders are Scalable Vision Learners（MAE）的编码器 - 解码器结构集成到 SSAST 中。我们发现 MAE 预训练可以相较于当前的音频预训练策略，在常规模型和输入尺寸下提供 3 倍的加速和 2 倍的内存使用率降低。在下游任务的微调中，我们发现我们的方法比 SSAST 在各种下游任务中表现更优。我们进一步对预训练的不同策略进行了全面的评估，并探讨了视觉和音频领域之间 MAE 风格预训练的不同之处。