Recent general-purpose audio representations show state-of-the-art
performance on various audio tasks. These representations are pre-trained by
self-supervised learning methods that create training signals from the input.
For example, typical audio contrastive learning uses temporal relationships
among input sounds to create training signals, whereas some methods use a
difference among input views created by data augmentations. However, these
training signals do not provide information derived from the intact input
sound, which we think is suboptimal for learning representation that describes
the input as it is.
In this paper, we seek to learn audio representations from the input itself
as supervision using a pretext task of auto-encoding of masked spectrogram
patches, Masked Spectrogram Modeling (MSM, a variant of Masked Image Modeling
applied to audio spectrogram). To implement MSM, we use Masked Autoencoders
(MAE), an image self-supervised learning method. MAE learns to efficiently
encode the small number of visible patches into latent representations to carry
essential information for reconstructing a large number of masked patches.
While training, MAE minimizes the reconstruction error, which uses the input as
training signal, consequently achieving our goal.
We conducted experiments on our MSM using MAE (MSM-MAE) models under the
evaluation benchmark of the HEAR 2021 NeurIPS Challenge. Our MSM-MAE models
outperformed the HEAR 2021 Challenge results on seven out of 15 tasks (e.g.,
accuracies of 73.4% on CREMA-D and 85.8% on LibriCount), while showing top
performance on other tasks where specialized models perform better. We also
investigate how the design choices of MSM-MAE impact the performance and
conduct qualitative analysis of visualization outcomes to gain an understanding
of learned representations. We make our code available online.

本文提出了一种新的通过自编码集成原始音频数据的方法：Masked Spectrogram Modeling（MSM），并使用 Masked Autoencoders（MAE）进行自监督学习，这种方法在 HEAR 2021 NeurIPS Challenge 中取得了比传统方法更好的结果。