The current popular methods for video object segmentation (VOS) implement
feature matching through several hand-crafted modules that separately perform
feature extraction and matching. However, the above hand-crafted designs
empirically cause insufficient target interaction, thus limiting the dynamic
target-aware feature learning in VOS. To tackle these limitations, this paper
presents a scalable Simplified VOS (SimVOS) framework to perform joint feature
extraction and matching by leveraging a single transformer backbone.
Specifically, SimVOS employs a scalable ViT backbone for simultaneous feature
extraction and matching between query and reference features. This design
enables SimVOS to learn better target-ware features for accurate mask
prediction. More importantly, SimVOS could directly apply well-pretrained ViT
backbones (e.g., MAE) for VOS, which bridges the gap between VOS and
large-scale self-supervised pre-training. To achieve a better performance-speed
trade-off, we further explore within-frame attention and propose a new token
refinement module to improve the running speed and save computational cost.
Experimentally, our SimVOS achieves state-of-the-art results on popular video
object segmentation benchmarks, i.e., DAVIS-2017 (88.0% J&F), DAVIS-2016 (92.9%
J&F) and YouTube-VOS 2019 (84.2% J&F), without applying any synthetic video or
BL30K pre-training used in previous VOS approaches.

本研究提出了一种可扩展的简化 VOS（SimVOS）框架，通过利用单个转换器主干进行联合特征提取和匹配，从而实现了针对准确遮罩预测的更好的目标感知特征学习。实验证明，SimVOS 在流行的视频对象分割基准测试中取得了最先进的结果，即 DAVIS-2017（88.0％J＆F），DAVIS-2016（92.9％J＆F）和 YouTube-VOS 2019（84.2％J＆F），而不使用任何先前 VOS 方法中使用的合成视频或 BL30K 预训练。