We propose a method named AudioFormer, which learns audio feature
representations through the acquisition of discrete acoustic codes and
subsequently fine-tunes them for audio classification tasks. Initially, we
introduce a novel perspective by considering the audio classification task as a
form of natural language understanding (NLU). Leveraging an existing neural
audio codec model, we generate discrete acoustic codes and utilize them to
train a masked language model (MLM), thereby obtaining audio feature
representations. Furthermore, we pioneer the integration of a
\textbf{M}ulti-\textbf{P}ositive sample \textbf{C}ontrastive (MPC) learning
approach. This method enables the learning of joint representations among
multiple discrete acoustic codes within the same audio input. In our
experiments, we treat discrete acoustic codes as textual data and train a
masked language model using a cloze-like methodology, ultimately deriving
high-quality audio representations. Notably, the MPC learning technique
effectively captures collaborative representations among distinct positive
samples. Our research outcomes demonstrate that AudioFormer attains
significantly improved performance compared to prevailing monomodal audio
classification models across multiple datasets, and even outperforms
audio-visual multimodal classification models on select datasets. Specifically,
our approach achieves remarkable results on datasets including AudioSet (2M,
20K), and FSD50K, with performance scores of 53.9, 45.1, and 65.6,
respectively. We have openly shared both the code and models:
https://github.com/LZH-0225/AudioFormer.git.

通过收集离散声学代码并进行微调，AudioFormer 对音频分类任务提出了一种学习音频特征表示的方法。采用已有的神经音频编解码模型，生成离散声学代码并利用它们来训练掩码语言模型，从而获取音频特征表示。此外，通过采用多正样本对比学习方法，实现了多个离散声学代码在同一音频输入中的联合表示学习。实验结果表明，与传统单模音频分类模型相比，AudioFormer 在多个数据集上取得了显著提升，在某些数据集上甚至超越了音频 - 视觉多模分类模型的性能。