Contemporary Video Object Segmentation (VOS) approaches typically consist
stages of feature extraction, matching, memory management, and multiple objects
aggregation. Recent advanced models either employ a discrete modeling for these
components in a sequential manner, or optimize a combined pipeline through
substructure aggregation. However, these existing explicit staged approaches
prevent the VOS framework from being optimized as a unified whole, leading to
the limited capacity and suboptimal performance in tackling complex videos. In
this paper, we propose OneVOS, a novel framework that unifies the core
components of VOS with All-in-One Transformer. Specifically, to unify all
aforementioned modules into a vision transformer, we model all the features of
frames, masks and memory for multiple objects as transformer tokens, and
integrally accomplish feature extraction, matching and memory management of
multiple objects through the flexible attention mechanism. Furthermore, a
Unidirectional Hybrid Attention is proposed through a double decoupling of the
original attention operation, to rectify semantic errors and ambiguities of
stored tokens in OneVOS framework. Finally, to alleviate the storage burden and
expedite inference, we propose the Dynamic Token Selector, which unveils the
working mechanism of OneVOS and naturally leads to a more efficient version of
OneVOS. Extensive experiments demonstrate the superiority of OneVOS, achieving
state-of-the-art performance across 7 datasets, particularly excelling in
complex LVOS and MOSE datasets with 70.1% and 66.4% $J \& F$ scores, surpassing
previous state-of-the-art methods by 4.2% and 7.0%, respectively. And our code
will be available for reproducibility and further research.

在这篇论文中，我们提出了 OneVOS，这是一个使用全新的 All-in-One Transformer 将 VOS 核心组件统一起来的框架。我们通过将帧、掩码和多个对象的所有特征建模为 Transformer tokens，通过灵活的注意力机制整体完成多个对象的特征提取、匹配和记忆管理。此外，我们通过原始注意力操作的两个解耦合来提出了一种单向混合注意力，以更正 OneVOS 框架中存储的 token 的语义错误和歧义。最后，为了减轻存储负担并加快推理速度，我们提出了动态 Token 选择器，该选择器揭示了 OneVOS 的工作机制，并自然而然地导致了更高效的 OneVOS 版本。广泛的实验证明了 OneVOS 的优越性，在七个数据集上实现了最先进的性能，特别是在复杂的 LVOS 和 MOSE 数据集上，J＆F 得分分别为 70.1％和 66.4％，超过了先前最先进的方法 4.2％和 7.0％。我们的代码可供再现性和进一步研究。