Audiovisual segmentation (AVS) is a challenging task that aims to segment
visual objects in videos based on their associated acoustic cues. With multiple
sound sources involved, establishing robust correspondences between audio and
visual contents poses unique challenges due to its (1) intricate entanglement
across sound sources and (2) frequent shift among sound events. Assuming sound
events occur independently, the multi-source semantic space (which encompasses
all possible semantic categories) can be viewed as the Cartesian product of
single-source sub-spaces. This motivates us to decompose the multi-source audio
semantics into single-source semantics, allowing for more effective interaction
with visual content. Specifically, we propose a semantic decomposition method
based on product quantization, where the multi-source semantics can be
decomposed and represented by several quantized single-source semantics.
Furthermore, we introduce a global-to-local quantization mechanism that
distills knowledge from stable global (clip-level) features into local
(frame-level) ones to handle the constant shift of audio semantics. Extensive
experiments demonstrate that semantically quantized and decomposed audio
representation significantly improves AVS performance, e.g., +21.2% mIoU on the
most challenging AVS-Semantic benchmark.

基于产品量化的方法可以将多源音频语义分解为多个量化的单源音频语义，通过将稳定的全局特征知识提炼到局部特征上来处理音频语义的频繁转变，并在音频视觉分割方面显著提高性能。