The aim of audio-visual segmentation (AVS) is to precisely differentiate
audible objects within videos down to the pixel level. Traditional approaches
often tackle this challenge by combining information from various modalities,
where the contribution of each modality is implicitly or explicitly modeled.
Nevertheless, the interconnections between different modalities tend to be
overlooked in audio-visual modeling. In this paper, inspired by the human
ability to mentally simulate the sound of an object and its visual appearance,
we introduce a bidirectional generation framework. This framework establishes
robust correlations between an object's visual characteristics and its
associated sound, thereby enhancing the performance of AVS. To achieve this, we
employ a visual-to-audio projection component that reconstructs audio features
from object segmentation masks and minimizes reconstruction errors. Moreover,
recognizing that many sounds are linked to object movements, we introduce an
implicit volumetric motion estimation module to handle temporal dynamics that
may be challenging to capture using conventional optical flow methods. To
showcase the effectiveness of our approach, we conduct comprehensive
experiments and analyses on the widely recognized AVSBench benchmark. As a
result, we establish a new state-of-the-art performance level in the AVS
benchmark, particularly excelling in the challenging MS3 subset which involves
segmenting multiple sound sources. To facilitate reproducibility, we plan to
release both the source code and the pre-trained model.

通过建立视觉特征与声音的鲁棒相关性，利用双向生成框架实现音频 - 视觉分割的改进性能，在 AVS 基准测试中取得新的最先进表现水平，并发布源代码与预训练模型。