Recently, video object segmentation (VOS) referred by multi-modal signals,
e.g., language and audio, has evoked increasing attention in both industry and
academia. It is challenging for exploring the semantic alignment within
modalities and the visual correspondence across frames. However, existing
methods adopt separate network architectures for different modalities, and
neglect the inter-frame temporal interaction with references. In this paper, we
propose MUTR, a Multi-modal Unified Temporal transformer for Referring video
object segmentation. With a unified framework for the first time, MUTR adopts a
DETR-style transformer and is capable of segmenting video objects designated by
either text or audio reference. Specifically, we introduce two strategies to
fully explore the temporal relations between videos and multi-modal signals.
Firstly, for low-level temporal aggregation before the transformer, we enable
the multi-modal references to capture multi-scale visual cues from consecutive
video frames. This effectively endows the text or audio signals with temporal
knowledge and boosts the semantic alignment between modalities. Secondly, for
high-level temporal interaction after the transformer, we conduct inter-frame
feature communication for different object embeddings, contributing to better
object-wise correspondence for tracking along the video. On Ref-YouTube-VOS and
AVSBench datasets with respective text and audio references, MUTR achieves
+4.2% and +4.2% J&F improvements to state-of-the-art methods, demonstrating our
significance for unified multi-modal VOS. Code is released at
this https URL

本文提出了 MUTR，通过统一框架和两种策略，实现了同时支持文本和音频引用的视频对象分割，实现了视频内部各帧的时序交互，从而提高了语义对齐和目标对应的准确性。MUTR 在多个数据集上都达到了更好的 J＆F 性能。