The combination of audio and vision has long been a topic of interest in the
multi-modal community. Recently, a new audio-visual segmentation (AVS) task has
been introduced, aiming to locate and segment the sounding objects in a given
video. This task demands audio-driven pixel-level scene understanding for the
first time, posing significant challenges. In this paper, we propose
AVSegFormer, a novel framework for AVS tasks that leverages the transformer
architecture. Specifically, we introduce audio queries and learnable queries
into the transformer decoder, enabling the network to selectively attend to
interested visual features. Besides, we present an audio-visual mixer, which
can dynamically adjust visual features by amplifying relevant and suppressing
irrelevant spatial channels. Additionally, we devise an intermediate mask loss
to enhance the supervision of the decoder, encouraging the network to produce
more accurate intermediate predictions. Extensive experiments demonstrate that
AVSegFormer achieves state-of-the-art results on the AVS benchmark. The code is
available at this https URL

本文提出了一种基于 transformer 架构的音视频分割（AVS）框架 AVSegFormer，该框架引入了音频查询和可学习查询，利用注意力机制实现选择性关注有关的视觉特征，并使用中间 mask 损失增强了解码器的监督，有效解决了音视频分割任务中的重要挑战，实验结果表明，AVSegFormer 在 AVS 基准测试中取得了最佳性能。