The task of audio-visual sound source localization has been well studied
under constrained scenes, where the audio recordings are clean. However, in
real-world scenarios, audios are usually contaminated by off-screen sound and
background noise. They will interfere with the procedure of identifying desired
sources and building visual-sound connections, making previous studies
non-applicable. In this work, we propose the Interference Eraser (IEr)
framework, which tackles the problem of audio-visual sound source localization
in the wild. The key idea is to eliminate the interference by redefining and
carving discriminative audio representations. Specifically, we observe that the
previous practice of learning only a single audio representation is
insufficient due to the additive nature of audio signals. We thus extend the
audio representation with our Audio-Instance-Identifier module, which clearly
distinguishes sounding instances when audio signals of different volumes are
unevenly mixed. Then we erase the influence of the audible but off-screen
sounds and the silent but visible objects by a Cross-modal Referrer module with
cross-modality distillation. Quantitative and qualitative evaluations
demonstrate that our proposed framework achieves superior results on sound
localization tasks, especially under real-world scenarios. Code is available at
this https URL

本文提出了 Interference Eraser（IEr）框架，用于处理复杂的真实场景中的音频 - 视觉声源定位问题。通过增强音频表示和采用跨模态引用模块的跨模态蒸馏，消除混杂声音和噪音对音频 - 视觉连接的干扰，在定量和定性评估中，我们的框架取得了在声音定位任务上卓越的结果。