The objective of the sound source localization task is to enable machines to
detect the location of sound-making objects within a visual scene. While the
audio modality provides spatial cues to locate the sound source, existing
approaches only use audio as an auxiliary role to compare spatial regions of
the visual modality. Humans, on the other hand, utilize both audio and visual
modalities as spatial cues to locate sound sources. In this paper, we propose
an audio-visual spatial integration network that integrates spatial cues from
both modalities to mimic human behavior when detecting sound-making objects.
Additionally, we introduce a recursive attention network to mimic human
behavior of iterative focusing on objects, resulting in more accurate attention
regions. To effectively encode spatial information from both modalities, we
propose audio-visual pair matching loss and spatial region alignment loss. By
utilizing the spatial cues of audio-visual modalities and recursively focusing
objects, our method can perform more robust sound source localization.
Comprehensive experimental results on the Flickr SoundNet and VGG-Sound Source
datasets demonstrate the superiority of our proposed method over existing
approaches. Our code is available at: this https URL

提出了一种音频 - 视觉空间整合网络，利用音频和视觉模态的空间线索来模仿人类在检测制造声音的对象时的行为，并引入了递归注意网络来递归地专注于对象，从而形成更准确的注意区域，通过利用音频 - 视觉模态的空间线索和递归地关注对象，我们的方法可以实现更稳健的声源定位，全面的实验结果表明了该方法优于现有方法。