We propose an Explicit Conditional Multimodal Variational Auto-Encoder
(ECMVAE) for audio-visual segmentation (AVS), aiming to segment sound sources
in the video sequence. Existing AVS methods focus on implicit feature fusion
strategies, where models are trained to fit the discrete samples in the
dataset. With a limited and less diverse dataset, the resulting performance is
usually unsatisfactory. In contrast, we address this problem from an effective
representation learning perspective, aiming to model the contribution of each
modality explicitly. Specifically, we find that audio contains critical
category information of the sound producers, and visual data provides candidate
sound producer(s). Their shared information corresponds to the target sound
producer(s) shown in the visual data. In this case, cross-modal shared
representation learning is especially important for AVS. To achieve this, our
ECMVAE factorizes the representations of each modality with a modality-shared
representation and a modality-specific representation. An orthogonality
constraint is applied between the shared and specific representations to
maintain the exclusive attribute of the factorized latent code. Further, a
mutual information maximization regularizer is introduced to achieve extensive
exploration of each modality. Quantitative and qualitative evaluations on the
AVSBench demonstrate the effectiveness of our approach, leading to a new
state-of-the-art for AVS, with a 3.84 mIOU performance leap on the challenging
MS3 subset for multiple sound source segmentation.

通过提出的明确条件多模态变分自编码器（ECMVAE）来进行音频 - 视觉分割（AVS），从有效的表示学习的角度解决现有 AVS 方法中的问题，该方法的关键词包括明确条件多模态变分自编码器、音频 - 视觉分割、表示学习、交叉模态共享表示和音源分割。