Despite the recent success of video self-supervised learning, there is much still to be understood about their generalization capability. In this paper, we investigate how sensitive video self-supervised learning is to the currently used benchmark convention and whether methods generalize beyond the canonical evaluation setting. We do this across four different factors of sensitivity: domain, samples, actions and task. Our comprehensive set of over 500 experiments, which encompasses 7 video datasets, 9 self-supervised methods and 6 video understanding tasks, reveals that current benchmarks in video self-supervised learning are not a good indicator of generalization along these sensitivity factors. Further, we find that self-supervised methods considerably lag behind vanilla supervised pre-training, especially when domain shift is large and the amount of available downstream samples are low. From our analysis we distill the SEVERE-benchmark, a subset of our experiments, and discuss its implication for evaluating the generalizability of representations obtained by existing and future self-supervised video learning methods.

本文研究了自监督视频学习对当前常规基准的敏感性以及能否在规范化评估设置之外进行泛化。通过对七个视频数据集、九种自监督方法和六种视频理解任务进行超过500次实验，我们发现自监督方法在大规模领域转移和可用样本数量较少的情况下，特别是在当前基准中，远远落后于普通监督预训练。通过我们的研究，我们总结了一种子集 SEVERE-benchmark，并讨论了其对现有和未来自监督视频学习方法获得表征的泛化能力的评估的影响。

视频自监督学习中基准敏感性有多严重？