Audio-Visual Segmentation (AVS) aims to identify, at the pixel level, the
object in a visual scene that produces a given sound. Current AVS methods rely
on costly fine-grained annotations of mask-audio pairs, making them impractical
for scalability. To address this, we introduce unsupervised AVS, eliminating
the need for such expensive annotation. To tackle this more challenging
problem, we propose an unsupervised learning method, named Modality
Correspondence Alignment (MoCA), which seamlessly integrates off-the-shelf
foundation models like DINO, SAM, and ImageBind. This approach leverages their
knowledge complementarity and optimizes their joint usage for multi-modality
association. Initially, we estimate positive and negative image pairs in the
feature space. For pixel-level association, we introduce an audio-visual
adapter and a novel pixel matching aggregation strategy within the image-level
contrastive learning framework. This allows for a flexible connection between
object appearance and audio signal at the pixel level, with tolerance to
imaging variations such as translation and rotation. Extensive experiments on
the AVSBench (single and multi-object splits) and AVSS datasets demonstrate
that our MoCA outperforms strongly designed baseline methods and approaches
supervised counterparts, particularly in complex scenarios with multiple
auditory objects. Notably when comparing mIoU, MoCA achieves a substantial
improvement over baselines in both the AVSBench (S4: +17.24%; MS3: +67.64%) and
AVSS (+19.23%) audio-visual segmentation challenges.

通过无监督学习方法 MoCA，在像素级上将音频和视觉图像相互关联，实现音频视觉分割的目标，超过基线方法并在复杂情况下实现了显著的性能提升。