This paper presents a new self-supervised video representation learning
framework, ARVideo, which autoregressively predicts the next video token in a
tailored sequence order. Two key designs are included. First, we organize
autoregressive video tokens into clusters that span both spatially and
temporally, thereby enabling a richer aggregation of contextual information
compared to the standard spatial-only or temporal-only clusters. Second, we
adopt a randomized spatiotemporal prediction order to facilitate learning from
multi-dimensional data, addressing the limitations of a handcrafted
spatial-first or temporal-first sequence order. Extensive experiments establish
ARVideo as an effective paradigm for self-supervised video representation
learning. For example, when trained with the ViT-B backbone, ARVideo
competitively attains 81.2% on Kinetics-400 and 70.9% on Something-Something
V2, which are on par with the strong benchmark set by VideoMAE. Importantly,
ARVideo also demonstrates higher training efficiency, i.e., it trains 14%
faster and requires 58% less GPU memory compared to VideoMAE.

该论文介绍了一种新的自我监督视频表示学习框架 ARVideo，其中通过自回归方式预测下一个视频令牌，将自回归视频令牌按照时空间隔进行聚类，并采用随机化时空预测顺序来提高学习效果，实验证明 ARVideo 是一种有效的自我监督视频表示学习范式，具有更高的训练效率。

ARVideo：用于自监督视频表示学习的自回归预训练

ARVideo: Autoregressive Pretraining for Self-Supervised Video  Representation Learning

This paper proposes a novel pretext task to address the self-supervised video
representation learning problem. Specifically, given an unlabeled video clip,
we compute a series of spatio-temporal statistical summaries, such as the
spatial location and dominant direction of the largest motion, the spatial
location and dominant color of the largest color diversity along the temporal
axis, etc. Then a neural network is built and trained to yield the statistical
summaries given the video frames as inputs. In order to alleviate the learning
difficulty, we employ several spatial partitioning patterns to encode rough
spatial locations instead of exact spatial Cartesian coordinates. Our approach
is inspired by the observation that human visual system is sensitive to rapidly
changing contents in the visual field, and only needs impressions about rough
spatial locations to understand the visual contents. To validate the
effectiveness of the proposed approach, we conduct extensive experiments with
four 3D backbone networks, i.e., C3D, 3D-ResNet, R(2+1)D and S3D-G. The results
show that our approach outperforms the existing approaches across these
backbone networks on four downstream video analysis tasks including action
recognition, video retrieval, dynamic scene recognition, and action similarity
labeling. The source code is publicly available at:
this https URL

本文旨在提出一种自监督视频表示学习的新型先验任务，通过计算一系列时空统计摘要信息，利用神经网络训练来产生摘要信息，采用多种空间分区模式进行粗略的空间位置编码方法来缓解学习难度，在四个 3D 骨干网络上的实验结果表明，该方法优于现有方法在视频分析任务上的性能表现包括动作识别、视频检索、动态场景识别和动作相似性标签。

通过发掘时空统计信息进行自监督视频表示学习

Self-supervised Video Representation Learning by Uncovering  Spatio-temporal Statistics

This paper addresses the problem of self-supervised video representation
learning from a new perspective -- by video pace prediction. It stems from the
observation that human visual system is sensitive to video pace, e.g., slow
motion, a widely used technique in film making. Specifically, given a video
played in natural pace, we randomly sample training clips in different paces
and ask a neural network to identify the pace for each video clip. The
assumption here is that the network can only succeed in such a pace reasoning
task when it understands the underlying video content and learns representative
spatio-temporal features. In addition, we further introduce contrastive
learning to push the model towards discriminating different paces by maximizing
the agreement on similar video content. To validate the effectiveness of the
proposed method, we conduct extensive experiments on action recognition and
video retrieval tasks with several alternative network architectures.
Experimental evaluations show that our approach achieves state-of-the-art
performance for self-supervised video representation learning across different
network architectures and different benchmarks. The code and pre-trained models
are available at this https URL

本文提出了一种新的自监督视频表示学习方法，即通过视频速度预测的方法来引导网络学习空间 - 时间特征，通过对相似视频内容最大化协同一致性来使模型区分不同速度的视频。实验表明，该方法在不同网络结构和数据集上均取得了现有最佳结果。