Audio-guided Video Object Segmentation (A-VOS) and Referring Video Object
Segmentation (R-VOS) are two highly-related tasks, which both aim to segment
specific objects from video sequences according to user-provided expression
prompts. However, due to the challenges in modeling representations for
different modalities, contemporary methods struggle to strike a balance between
interaction flexibility and high-precision localization and segmentation. In
this paper, we address this problem from two perspectives: the alignment
representation of audio and text and the deep interaction among audio, text,
and visual features. First, we propose a universal architecture, the Expression
Prompt Collaboration Transformer, herein EPCFormer. Next, we propose an
Expression Alignment (EA) mechanism for audio and text expressions. By
introducing contrastive learning for audio and text expressions, the proposed
EPCFormer realizes comprehension of the semantic equivalence between audio and
text expressions denoting the same objects. Then, to facilitate deep
interactions among audio, text, and video features, we introduce an
Expression-Visual Attention (EVA) mechanism. The knowledge of video object
segmentation in terms of the expression prompts can seamlessly transfer between
the two tasks by deeply exploring complementary cues between text and audio.
Experiments on well-recognized benchmarks demonstrate that our universal
EPCFormer attains state-of-the-art results on both tasks. The source code of
EPCFormer will be made publicly available at
this https URL

基于音频和文本之间的语义等价理解和音频、文本、视频特征之间的深度交互，我们提出了一种基于表达协作 Transformer（EPCFormer）的通用架构，并引入了表达对齐机制和表达 - 视觉注意机制，以实现音频和文本对象之间的高准确定位和分割，从而在两个相关任务上取得了最新的结果。