The ability to accurately recognize, localize and separate sound sources is
fundamental to any audio-visual perception task. Historically, these abilities
were tackled separately, with several methods developed independently for each
task. However, given the interconnected nature of source localization,
separation, and recognition, independent models are likely to yield suboptimal
performance as they fail to capture the interdependence between these tasks. To
address this problem, we propose a unified audio-visual learning framework
(dubbed OneAVM) that integrates audio and visual cues for joint localization,
separation, and recognition. OneAVM comprises a shared audio-visual encoder and
task-specific decoders trained with three objectives. The first objective
aligns audio and visual representations through a localized audio-visual
correspondence loss. The second tackles visual source separation using a
traditional mix-and-separate framework. Finally, the third objective reinforces
visual feature separation and localization by mixing images in pixel space and
aligning their representations with those of all corresponding sound sources.
Extensive experiments on MUSIC, VGG-Instruments, VGG-Music, and VGGSound
datasets demonstrate the effectiveness of OneAVM for all three tasks,
audio-visual source localization, separation, and nearest neighbor recognition,
and empirically demonstrate a strong positive transfer between them.

该研究提出了一种名为 OneAVM 的联合学习框架，该框架可以用于音频 - 视频源定位、分离和识别任务，其中共享的音频 - 视频编码器和任务特定的解码器是通过三个目标进行训练，包括本地化的音频 - 视觉对应丢失、视觉源分离和选择和用于强化视觉特征分离和定位的像素空间混合。经过在多个数据集上的广泛实验，证明了 OneAVM 的有效性，并在音频 - 视觉源定位、分离和最近邻识别任务之间展现了强大的正向转移。