Sound source localization is a typical and challenging task that predicts the
location of sound sources in a video. Previous single-source methods mainly
used the audio-visual association as clues to localize sounding objects in each
image. Due to the mixed property of multiple sound sources in the original
space, there exist rare multi-source approaches to localizing multiple sources
simultaneously, except for one recent work using a contrastive random walk in
the graph with images and separated sound as nodes. Despite their promising
performance, they can only handle a fixed number of sources, and they cannot
learn compact class-aware representations for individual sources. To alleviate
this shortcoming, in this paper, we propose a novel audio-visual grouping
network, namely AVGN, that can directly learn category-wise semantic features
for each source from the input audio mixture and image to localize multiple
sources simultaneously. Specifically, our AVGN leverages learnable audio-visual
class tokens to aggregate class-aware source features. Then, the aggregated
semantic features for each source can be used as guidance to localize the
corresponding visual regions. Compared to existing multi-source methods, our
new framework can localize a flexible number of sources and disentangle
category-aware audio-visual representations for individual sound sources. We
conduct extensive experiments on MUSIC, VGGSound-Instruments, and VGG-Sound
Sources benchmarks. The results demonstrate that the proposed AVGN can achieve
state-of-the-art sounding object localization performance on both single-source
and multi-source scenarios. Code is available at
https://github.com/stoneMo/AVGN.

本文提出了一种 AVGN 网络，通过可学习的音频 - 视觉类别权重直接学习每个音频源的语义特征，可以同时定位多个音频源，达到了先进的音响目标定位效果。

用于混音声源定位的音视频分组网络

Audio-Visual Grouping Network for Sound Localization from Mixtures

Leveraging temporal synchronization and association within sight and sound is
an essential step towards robust localization of sounding objects. To this end,
we propose a space-time memory network for sounding object localization in
videos. It can simultaneously learn spatio-temporal attention over both
uni-modal and cross-modal representations from audio and visual modalities. We
show and analyze both quantitatively and qualitatively the effectiveness of
incorporating spatio-temporal learning in localizing audio-visual objects. We
demonstrate that our approach generalizes over various complex audio-visual
scenes and outperforms recent state-of-the-art methods.

提出了一种利用时间同步和关联的声音和视觉信息的网络，用于视频中声音对象的本地化