In this paper, we study masked autoencoder (MAE) pretraining on videos for
matching-based downstream tasks, including visual object tracking (VOT) and
video object segmentation (VOS). A simple extension of MAE is to randomly mask
out frame patches in videos and reconstruct the frame pixels. However, we find
that this simple baseline heavily relies on spatial cues while ignoring
temporal relations for frame reconstruction, thus leading to sub-optimal
temporal matching representations for VOT and VOS. To alleviate this problem,
we propose DropMAE, which adaptively performs spatial-attention dropout in the
frame reconstruction to facilitate temporal correspondence learning in videos.
We show that our DropMAE is a strong and efficient temporal matching learner,
which achieves better finetuning results on matching-based tasks than the
ImageNetbased MAE with 2X faster pre-training speed. Moreover, we also find
that motion diversity in pre-training videos is more important than scene
diversity for improving the performance on VOT and VOS. Our pre-trained DropMAE
model can be directly loaded in existing ViT-based trackers for fine-tuning
without further modifications. Notably, DropMAE sets new state-of-the-art
performance on 8 out of 9 highly competitive video tracking and segmentation
datasets. Our code and pre-trained models are available at
this https URL

该研究旨在研究在视频上使用掩蔽自动编码器（MAE）进行预训练，从而实现针对视觉对象跟踪（VOT）和视频对象分割（VOS）等基于匹配的下游任务。所提出的 DropMAE 是一种强大的高效的时间匹配学习器，在与 ImageNet-based MAE 相比较有着相同的表现，且其预培训速度更快。同时，该研究发现，先验于 DropMAE 中应用的视频的运动多样性比场景多样性对于提高 VOT 和 VOS 的性能更加重要。