Visual sound source localization poses a significant challenge in identifying
the semantic region of each sounding source within a video. Existing
self-supervised and weakly supervised source localization methods struggle to
accurately distinguish the semantic regions of each sounding object,
particularly in multi-source mixtures. These methods often rely on audio-visual
correspondence as guidance, which can lead to substantial performance drops in
complex multi-source localization scenarios. The lack of access to individual
source sounds in multi-source mixtures during training exacerbates the
difficulty of learning effective audio-visual correspondence for localization.
To address this limitation, in this paper, we propose incorporating the text
modality as an intermediate feature guide using tri-modal joint embedding
models (e.g., AudioCLIP) to disentangle the semantic audio-visual source
correspondence in multi-source mixtures. Our framework, dubbed T-VSL, begins by
predicting the class of sounding entities in mixtures. Subsequently, the
textual representation of each sounding source is employed as guidance to
disentangle fine-grained audio-visual source correspondence from multi-source
mixtures, leveraging the tri-modal AudioCLIP embedding. This approach enables
our framework to handle a flexible number of sources and exhibits promising
zero-shot transferability to unseen classes during test time. Extensive
experiments conducted on the MUSIC, VGGSound, and VGGSound-Instruments datasets
demonstrate significant performance improvements over state-of-the-art methods.

我们提出了一种利用 Tri-modal joint embedding 模型通过文本模态作为中间特征引导，在多源混合中分离语义音视源对应关系的 T-VSL 框架，该方法在训练期间通过预测混合中声音实体的类来引导细粒度的音视源对应关系的分离，展现了在测试期间对未见过的类别具有有希望的零 - shot 迁移能力。在 MUSIC、VGGSound 和 VGGSound-Instruments 数据集上的大量实验证明了该方法相对于最先进方法的显著性能提升。