The Associating Objects with Transformers (AOT) framework has exhibited
exceptional performance in a wide range of complex scenarios for video object
tracking and segmentation. In this study, we convert the bounding boxes to
masks in reference frames with the help of the Segment Anything Model (SAM) and
Alpha-Refine, and then propagate the masks to the current frame, transforming
the task from Video Object Tracking (VOT) to video object segmentation (VOS).
Furthermore, we introduce MSDeAOT, a variant of the AOT series that
incorporates transformers at multiple feature scales. MSDeAOT efficiently
propagates object masks from previous frames to the current frame using two
feature scales of 16 and 8. As a testament to the effectiveness of our design,
we achieved the 1st place in the EPIC-KITCHENS TREK-150 Object Tracking
Challenge.

通过 Associating Objects with Transformers (AOT) 框架，将 bounding boxes 转换为 masks，通过多个 feature scales 的转换实现视频对象跟踪和分割，并在 EPIC-KITCHENS TREK-150 Object Tracking Challenge 中获得第一名。

2023 年浙江大学 ReLER 提交的 EPIC-KITCHEN 挑战：TREK-150 单目标追踪

ZJU ReLER Submission for EPIC-KITCHEN Challenge 2023: TREK-150 Single  Object Tracking

The Associating Objects with Transformers (AOT) framework has exhibited
exceptional performance in a wide range of complex scenarios for video object
segmentation. In this study, we introduce MSDeAOT, a variant of the AOT series
that incorporates transformers at multiple feature scales. Leveraging the
hierarchical Gated Propagation Module (GPM), MSDeAOT efficiently propagates
object masks from previous frames to the current frame using a feature scale
with a stride of 16. Additionally, we employ GPM in a more refined feature
scale with a stride of 8, leading to improved accuracy in detecting and
tracking small objects. Through the implementation of test-time augmentations
and model ensemble techniques, we achieve the top-ranking position in the
EPIC-KITCHEN VISOR Semi-supervised Video Object Segmentation Challenge.

本研究介绍了 MSDeAOT，一种在多个特征尺度上运用 transformers 的 AOT 变体，利用分层的 GPM，以 stride 为 16 的特征尺度高效地从前一帧传播物体掩码至当前帧，同时在具有 stride 为 8 的更精细的特征尺度上应用 GPM，从而提高了检测和跟踪小物体的准确性。通过实现测试时间的数据增强和模型集成技术，我们在 EPIC-KITCHEN VISOR 半监督视频对象分割挑战中取得了排名第一的位置。

2023 EPIC-KITCHEN 挑战赛中的浙江大学 ReLER 提交：半监督视频目标分割

ZJU ReLER Submission for EPIC-KITCHEN Challenge 2023: Semi-Supervised  Video Object Segmentation

This paper investigates how to realize better and more efficient embedding
learning to tackle the semi-supervised video object segmentation under
challenging multi-object scenarios. The state-of-the-art methods learn to
decode features with a single positive object and thus have to match and
segment each target separately under multi-object scenarios, consuming multiple
times computing resources. To solve the problem, we propose an Associating
Objects with Transformers (AOT) approach to match and decode multiple objects
uniformly. In detail, AOT employs an identification mechanism to associate
multiple targets into the same high-dimensional embedding space. Thus, we can
simultaneously process multiple objects' matching and segmentation decoding as
efficiently as processing a single object. For sufficiently modeling
multi-object association, a Long Short-Term Transformer is designed for
constructing hierarchical matching and propagation. We conduct extensive
experiments on both multi-object and single-object benchmarks to examine AOT
variant networks with different complexities. Particularly, our R50-AOT-L
outperforms all the state-of-the-art competitors on three popular benchmarks,
i.e., YouTube-VOS (84.1% J&F), DAVIS 2017 (84.9%), and DAVIS 2016 (91.1%),
while keeping more than $3\times$ faster multi-object run-time. Meanwhile, our
AOT-T can maintain real-time multi-object speed on the above benchmarks. Based
on AOT, we ranked 1st in the 3rd Large-scale VOS Challenge.

本研究提出了基于 AOT 的嵌入式学习方法，通过将多个对象相互关联在同一高维度嵌入空间中，实现了同时处理多个对象的匹配和分割解码，具有更高的效率，并且在多个基准测试中均优于已有的 State-of-the-art 算法，是第三次大规模 VOS 挑战的第一名。