In this study, we initiate an exploration into video understanding by
introducing VideoChat, an end-to-end chat-centric video understanding system.
It integrates video foundation models and large language models via a learnable
neural interface, excelling in spatiotemporal reasoning, event localization,
and causal relationship inference. To instructively tune this system, we
propose a video-centric instruction dataset, composed of thousands of videos
matched with detailed descriptions and conversations. This dataset emphasizes
spatiotemporal reasoning and causal relationships, providing a valuable asset
for training chat-centric video understanding systems. Preliminary qualitative
experiments reveal our system's potential across a broad spectrum of video
applications and set the standard for future research. Access our code and data
at this https URL

本研究介绍了一种以视频为中心的语音理解系统 VideoChat，它通过可学习的神经接口将视频基础模型和大型语言模型集成起来，在时空推理、事件定位和因果关系推断等方面表现出色。通过使用视频为中心的指令数据集，我们为这个系统提供了有指导性的调整建议，这个数据集由成千上万个视频和详细的描述和会话组成，突出了时空推理和因果关系，为训练以聊天为中心的视频理解系统提供了有价值的资源。初步的定性实验揭示了我们的系统在广泛的视频应用中的潜力，并为未来的研究设置了标准。