The referring video object segmentation task (RVOS) involves segmentation of
a text-referred object instance in the frames of a given video. Due to the
complex nature of this multimodal task, which combines text reasoning, video
understanding, instance segmentation and tracking, existing approaches
typically rely on sophisticated pipelines in order to tackle it. In this paper,
we propose a simple Transformer-based approach to RVOS. Our framework, termed
Multimodal Tracking Transformer (MTTR), models the RVOS task as a sequence
prediction problem. Following recent advancements in computer vision and
natural language processing, MTTR is based on the realization that video and
text can be processed together effectively and elegantly by a single multimodal
Transformer model. MTTR is end-to-end trainable, free of text-related inductive
bias components and requires no additional mask-refinement post-processing
steps. As such, it simplifies the RVOS pipeline considerably compared to
existing methods. Evaluation on standard benchmarks reveals that MTTR
significantly outperforms previous art across multiple metrics. In particular,
MTTR shows impressive +5.7 and +5.0 mAP gains on the A2D-Sentences and
JHMDB-Sentences datasets respectively, while processing 76 frames per second.
In addition, we report strong results on the public validation set of
Refer-YouTube-VOS, a more challenging RVOS dataset that has yet to receive the
attention of researchers. The code to reproduce our experiments is available at
this https URL

本文提出了一种名为 Multimodal Tracking Transformer（MTTR）的基于 Transformer 的方法来实现 referring video object segmentation（RVOS）任务，该方法将视频和文本结合起来处理，不依赖于复杂的流水线，并且在标准基准测试中显著优于以前的方法。