Modeling and understanding time remains a challenge in contemporary video
understanding models. With language emerging as a key driver towards powerful
generalization, it is imperative for foundational video-language models to have
a sense of time. In this paper, we consider a specific aspect of temporal
understanding: consistency of time order as elicited by before/after relations.
We establish that six existing video-language models struggle to understand
even such simple temporal relations. We then question whether it is feasible to
equip these foundational models with temporal awareness without re-training
them from scratch. Towards this, we propose a temporal adaptation recipe on top
of one such model, VideoCLIP, based on post-pretraining on a small amount of
video-text data. We conduct a zero-shot evaluation of the adapted models on six
datasets for three downstream tasks which require a varying degree of time
awareness. We observe encouraging performance gains especially when the task
needs higher time awareness. Our work serves as a first step towards probing
and instilling a sense of time in existing video-language models without the
need for data and compute-intense training from scratch.

本文旨在向已有视频语言模型注入时间感，通过后预训练等方式实现，观察经过处理后的模型在时间感知任务中的性能提升情况，并在六个数据集上进行了零样本评估。