Training an effective video-and-language model intuitively requires multiple
frames as model inputs. However, it is unclear whether using multiple frames is
beneficial to downstream tasks, and if yes, whether the performance gain is
worth the drastically-increased computation and memory costs resulting from
using more frames. In this work, we explore single-frame models for
video-and-language learning. On a diverse set of video-and-language tasks
(including text-to-video retrieval and video question answering), we show the
surprising result that, with large-scale pre-training and a proper frame
ensemble strategy at inference time, a single-frame trained model that does not
consider temporal information can achieve better performance than existing
methods that use multiple frames for training. This result reveals the
existence of a strong "static appearance bias" in popular video-and-language
datasets. Therefore, to allow for a more comprehensive evaluation of
video-and-language models, we propose two new retrieval tasks based on existing
fine-grained action recognition datasets that encourage temporal modeling. Our
code is available at this https URL

本文探讨了视频与语言学习中基于单帧的模型的应用，结果表明在大规模预训练条件下，采用适当的帧合并策略的单帧训练模型在某些视频与语言任务方面表现更佳，并且作者提出了两个基于现有细粒度动作识别数据集的检索任务，以便更全面地评估视频与语言模型。