Video is an increasingly prominent and information-dense medium, yet it poses
substantial challenges for language models. A typical video consists of a
sequence of shorter segments, or shots, that collectively form a coherent
narrative. Each shot is analogous to a word in a sentence where multiple data
streams of information (such as visual and auditory data) must be processed
simultaneously. Comprehension of the entire video requires not only
understanding the visual-audio information of each shot but also requires that
the model links the ideas between each shot to generate a larger,
all-encompassing story. Despite significant progress in the field, current
works often overlook videos' more granular shot-by-shot semantic information.
In this project, we propose a family of efficient large language vision models
(LLVMs) to boost video summarization and captioning called Shotluck Holmes. By
leveraging better pretraining and data collection strategies, we extend the
abilities of existing small LLVMs from being able to understand a picture to
being able to understand a sequence of frames. Specifically, we show that
Shotluck Holmes achieves better performance than state-of-the-art results on
the Shot2Story video captioning and summary task with significantly smaller and
more computationally efficient models.

我们提出一种名为 Shotluck Holmes 的高效大型语言视觉模型 (LLVMs)，通过改进预训练和数据收集策略，扩展现有小型 LLVMs 的能力，从仅能理解一张图片到能够理解连续的帧序列，在 Shot2Story 视频字幕和摘要任务上取得比最先进结果更好的性能，同时模型尺寸更小、计算效率更高。