We introduce a new task, Video-and-Language Inference, for joint multimodal
understanding of video and text. Given a video clip with aligned subtitles as
premise, paired with a natural language hypothesis based on the video content,
a model needs to infer whether the hypothesis is entailed or contradicted by
the given video clip. A new large-scale dataset, named Violin
(VIdeO-and-Language INference), is introduced for this task, which consists of
95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours
of video. These video clips contain rich content with diverse temporal
dynamics, event shifts, and people interactions, collected from two sources:
(i) popular TV shows, and (ii) movie clips from YouTube channels. In order to
address our new multimodal inference task, a model is required to possess
sophisticated reasoning skills, from surface-level grounding (e.g., identifying
objects and characters in the video) to in-depth commonsense reasoning (e.g.,
inferring causal relations of events in the video). We present a detailed
analysis of the dataset and an extensive evaluation over many strong baselines,
providing valuable insights on the challenges of this new task.

本文介绍了一个新的任务 —— 视频与语言推理，用于联合多模态理解视频和文本，提出了一个名为 “Violin” 的大规模数据集并对其进行了分析，并对该任务进行了广泛评估。