Audio-visual representation learning aims to develop systems with human-like
perception by utilizing correlation between auditory and visual information.
However, current models often focus on a limited set of tasks, and
generalization abilities of learned representations are unclear. To this end,
we propose the AV-SUPERB benchmark that enables general-purpose evaluation of
unimodal audio/visual and bimodal fusion representations on 7 datasets covering
5 audio-visual tasks in speech and audio processing. We evaluate 5 recent
self-supervised models and show that none of these models generalize to all
tasks, emphasizing the need for future study on improving universal model
performance. In addition, we show that representations may be improved with
intermediate-task fine-tuning and audio event classification with AudioSet
serves as a strong intermediate task. We release our benchmark with evaluation
code and a model submission platform to encourage further research in
audio-visual learning.

音频 - 视觉表示学习，一种开发具有类似于人类感知的系统的方法，利用声音和视觉信息之间的相关性。然而，目前的模型往往专注于有限的任务集，并且对学习表示的泛化能力尚不清楚。因此，我们提出了 AV-SUPERB 基准，它在涵盖语音和音频处理中的 5 个音频 - 视觉任务的 7 个数据集上，能够对单模音频 / 视觉和双模融合表示进行通用评估。我们评估了 5 个最近的自监督模型，并表明这些模型都不能泛化到所有任务，强调了未来需要改进通用模型性能的研究的必要性。此外，我们表明通过中间任务微调和使用 AudioSet 进行音频事件分类可以改进表示。我们发布了我们的基准测试，提供了评估代码和模型提交平台，以鼓励进一步进行音频 - 视觉学习的研究。