Despite constituting 65% of all internet traffic in 2023, video content is
underrepresented in generative AI research. Meanwhile, recent large language
models (LLMs) have become increasingly integrated with capabilities in the
visual modality. Integrating video with LLMs is a natural next step, so how can
this gap be bridged? To advance video reasoning, we propose a new research
direction of VideoCOT on video keyframes, which leverages the multimodal
generative abilities of vision-language models to enhance video reasoning while
reducing the computational complexity of processing hundreds or thousands of
frames. We introduce VIP, an inference-time dataset that can be used to
evaluate VideoCOT, containing 1) a variety of real-life videos with keyframes
and corresponding unstructured and structured scene descriptions, and 2) two
new video reasoning tasks: video infilling and scene prediction. We benchmark
various vision-language models on VIP, demonstrating the potential to use
vision-language models and LLMs to enhance video chain of thought reasoning.

为了提高视频推理的能力和降低处理数百或数千帧的计算复杂度，我们提出了 VideoCOT 的新研究方向，旨在利用视觉语言模型的多模式生成能力对视频关键帧进行增强。我们引入了 VIP 数据集，其中包含各种现实生活视频和场景描述，以及两个新的视频推理任务：视频填充和场景预测，评估了各种视觉语言模型在 VIP 上的表现，证明了利用视觉语言模型和 LLM 提高视频链推理的潜力。