Never having seen an object and heard its sound simultaneously, can the model
still accurately localize its visual position from the input audio? In this
work, we concentrate on the Audio-Visual Localization and Segmentation tasks
but under the demanding zero-shot and few-shot scenarios. To achieve this goal,
different from existing approaches that mostly employ the
encoder-fusion-decoder paradigm to decode localization information from the
fused audio-visual feature, we introduce the encoder-prompt-decoder paradigm,
aiming to better fit the data scarcity and varying data distribution dilemmas
with the help of abundant knowledge from pre-trained models. Specifically, we
first propose to construct Semantic-aware Audio Prompt (SAP) to help the visual
foundation model focus on sounding objects, meanwhile, the semantic gap between
the visual and audio modalities is also encouraged to shrink. Then, we develop
a Correlation Adapter (ColA) to keep minimal training efforts as well as
maintain adequate knowledge of the visual foundation model. By equipping with
these means, extensive experiments demonstrate that this new paradigm
outperforms other fusion-based methods in both the unseen class and
cross-dataset settings. We hope that our work can further promote the
generalization study of Audio-Visual Localization and Segmentation in practical
application scenarios.

通过引入编码 - 提示 - 解码范式，本研究关注零样本和少样本情景下的视听定位和分割任务，并通过构建语义感知的音频提示和相关适配器，实现了在数据稀缺和数据分布差异方面的优越性能。

声音提示分割是可推广的音频 - 视觉源定位器

Prompting Segmentation with Sound is Generalizable Audio-Visual Source  Localizer

Segment Anything Model (SAM) has recently shown its powerful effectiveness in
visual segmentation tasks. However, there is less exploration concerning how
SAM works on audio-visual tasks, such as visual sound localization and
segmentation. In this work, we propose a simple yet effective audio-visual
localization and segmentation framework based on the Segment Anything Model,
namely AV-SAM, that can generate sounding object masks corresponding to the
audio. Specifically, our AV-SAM simply leverages pixel-wise audio-visual fusion
across audio features and visual features from the pre-trained image encoder in
SAM to aggregate cross-modal representations. Then, the aggregated cross-modal
features are fed into the prompt encoder and mask decoder to generate the final
audio-visual segmentation masks. We conduct extensive experiments on
Flickr-SoundNet and AVSBench datasets. The results demonstrate that the
proposed AV-SAM can achieve competitive performance on sounding object
localization and segmentation.

本文提出了基于 SAM 模型的简单而有效的音频 - 视觉定位和分割框架 AV-SAM，可以生成对应于音频的听觉对象掩模，实现像声音定位和分割等视听任务。