We evaluate existing foundation models video understanding capabilities using
a carefully designed experiment protocol consisting of three hallmark tasks
(action recognition, temporal localization, and spatiotemporal localization),
eight datasets well received by the community, and four adaptation methods
tailoring a foundation model (FM) for a downstream task. Moreover, we propose a
scalar VideoGLUE score (VGS) to measure an FMs efficacy and efficiency when
adapting to general video understanding tasks. Our main findings are as
follows. First, task-specialized models significantly outperform the six FMs
studied in this work, in sharp contrast to what FMs have achieved in natural
language and image understanding. Second,video-native FMs, whose pretraining
data contains the video modality, are generally better than image-native FMs in
classifying motion-rich videos, localizing actions in time, and understanding a
video of more than one action. Third, the video-native FMs can perform well on
video tasks under light adaptations to downstream tasks(e.g., freezing the FM
backbones), while image-native FMs win in full end-to-end finetuning. The first
two observations reveal the need and tremendous opportunities to conduct
research on video-focused FMs, and the last confirms that both tasks and
adaptation methods matter when it comes to the evaluation of FMs.

通过使用三种典型任务、八个备受社区欢迎的数据集和四种适配方法评估现有基础模型在视频理解方面的能力，并提出了一种衡量基础模型在适应一般视频理解任务时的有效性和效率的 VideoGLUE 分数 (VGS)。研究发现，与自然语言和图像理解中的基础模型成就形成鲜明对比，专门任务模型在本文所研究的六个基础模型中表现出显著的优势；包含视频模态的本机视频基础模型在分类富有动态的视频、时间定位动作和理解多个动作的视频方面通常表现更好；本机视频基础模型在轻微适配下（如冻结 FM 骨干）可以在视频任务上表现良好，而图像本机基础模型则在全面端到端的微调中胜出。前两项观察结果揭示了在视频基础模型方面进行研究的需求和巨大机遇，最后一项结果确认了在评估基础模型时任务和适配方法均起到重要作用。