A short clip of video may contain progression of multiple events and an interesting story line. A human need to capture both the event in every shot and associate them together to understand the story behind it. In this work, we present a new multi-shot video understanding benchmark Shot2Story20K with detailed shot-level captions and comprehensive video summaries. To facilitate better semantic understanding of videos, we provide captions for both visual signals and human narrations. We design several distinct tasks including single-shot video and narration captioning, multi-shot video summarization, and video retrieval with shot descriptions. Preliminary experiments show some challenges to generate a long and comprehensive video summary. Nevertheless, the generated imperfect summaries can already significantly boost the performance of existing video understanding tasks such as video question-answering, promoting an under-explored setting of video understanding with detailed summaries.

通过一个多镜头视频理解基准（Shot2Story20K）的详细镜头级标题和全面视频摘要，提供了一种更好的视频语义理解方法，包括可视化信号和人类叙述的标题，摘要，检索以及摘要生成，这将显著提升现有视频理解任务的性能，并促进了视频理解中对详细摘要的未开发研究领域。

Shot2Story20K：多割视频全面理解的新基准测试