Despite being (pre)trained on a massive amount of data, state-of-the-art
video-language alignment models are not robust to semantically-plausible
contrastive changes in the video captions. Our work addresses this by
identifying a broad spectrum of contrast misalignments, such as replacing
entities, actions, and flipping event order, which alignment models should be
robust against. To this end, we introduce the VideoCon, a video-language
alignment dataset constructed by a large language model that generates
plausible contrast video captions and explanations for differences between
original and contrast video captions. Then, a generative video-language model
is finetuned with VideoCon to assess video-language entailment and generate
explanations. Our VideoCon-based alignment model significantly outperforms
current models. It exhibits a 12-point increase in AUC for the video-language
alignment task on human-generated contrast captions. Finally, our model sets
new state of the art zero-shot performance in temporally-extensive
video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video
question answering (ATP-Hard). Moreover, our model shows superior performance
on novel videos and human-crafted captions and explanations. Our code and data
are available at this https URL

通过介绍 VideoCon，我们的研究在应对视频字幕中语义上合理的对比变化方面改进了视频 - 语言对齐模型的鲁棒性，提出了一个广泛的对比失配频谱，并通过一个大规模语言模型构建了基于对比视频字幕的 VideoCon 对齐数据集，我们的对齐模型在人为生成的对比字幕上在视频 - 语言对齐任务中的 AUC 指标上有了 12 个百分点的提升，同时在诸如文本到视频检索（SSv2-Temporal）和视频问答（ATP-Hard）等时间广泛的视频 - 语言任务中表现出了零 - shot 的最新性能。