Pre-training video transformers on extra large-scale datasets is generally
required to achieve premier performance on relatively small datasets. In this
paper, we show that video masked autoencoders (VideoMAE) are data-efficient
learners for self-supervised video pre-training (SSVP). We are inspired by the
recent ImageMAE and propose customized video tube masking with an extremely
high ratio. This simple design makes video reconstruction a more challenging
self-supervision task, thus encouraging extracting more effective video
representations during this pre-training process. We obtain three important
findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90%
to 95%) still yields favorable performance of VideoMAE. The temporally
redundant video content enables a higher masking ratio than that of images. (2)
VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k
videos) without using any extra data. (3) VideoMAE shows that data quality is
more important than data quantity for SSVP. Domain shift between pre-training
and target datasets is an important issue. Notably, our VideoMAE with the
vanilla ViT can achieve 87.4% on Kinetics-400, 75.4% on Something-Something V2,
91.3% on UCF101, and 62.6% on HMDB51, without using any extra data. Code is
available at this https URL

本文研究了使用 VideoMAE 进行自主监督视频预训练 (SSVP) 的数据高效性问题，并通过适当的视频屏蔽达到了良好的表现，进而证明数据质量对 SSVP 更加重要。