Benefiting from the advancements in large language models and cross-modal
alignment, existing multi-modal video understanding methods have achieved
prominent performance in offline scenario. However, online video streams, as
one of the most common media forms in the real world, have seldom received
attention. Compared to offline videos, the 'dynamic' nature of online video
streams poses challenges for the direct application of existing models and
introduces new problems, such as the storage of extremely long-term
information, interaction between continuous visual content and 'asynchronous'
user questions. Therefore, in this paper we present Flash-VStream, a
video-language model that simulates the memory mechanism of human. Our model is
able to process extremely long video streams in real-time and respond to user
queries simultaneously. Compared to existing models, Flash-VStream achieves
significant reductions in inference latency and VRAM consumption, which is
intimately related to performing understanding of online streaming video. In
addition, given that existing video understanding benchmarks predominantly
concentrate on offline scenario, we propose VStream-QA, a novel question
answering benchmark specifically designed for online video streaming
understanding. Comparisons with popular existing methods on the proposed
benchmark demonstrate the superiority of our method for such challenging
setting. To verify the generalizability of our approach, we further evaluate it
on existing video understanding benchmarks and achieves state-of-the-art
performance in offline scenarios as well. All code, models, and datasets are
available at the this https URL

在这篇论文中，我们介绍了一种名为 Flash-VStream 的视频语言模型，它模拟了人类的记忆机制，能够实时处理极长的视频流并同时对用户查询进行响应。与现有模型相比，Flash-VStream 在推理延迟和 VRAM 消耗方面实现了显著的减少，并在在线视频流理解领域表现出了优异的性能。我们还提出了 VStream-QA，一个专门为在线视频流理解设计的新型问答基准，与现有方法在该基准上的比较结果显示了我们方法在这个具有挑战性场景中的优势。此外，我们的方法在离线场景中也取得了最先进的性能。