This work proposes TimeChat, a time-sensitive multimodal large language model
specifically designed for long video understanding. Our model incorporates two
key architectural contributions: (1) a timestamp-aware frame encoder that binds
visual content with the timestamp of each frame, and (2) a sliding video
Q-Former that produces a video token sequence of varying lengths to accommodate
videos of various durations. Additionally, we construct an instruction-tuning
dataset, encompassing 6 tasks and a total of 125K instances, to further enhance
TimeChat's instruction-following performance. Experiment results across various
video understanding tasks, such as dense captioning, temporal grounding, and
highlight detection, demonstrate TimeChat's strong zero-shot temporal
localization and reasoning capabilities. For example, it achieves +9.2 F1 score
and +2.8 CIDEr on YouCook2, +5.8 HIT@1 on QVHighlights, and +27.5 R@1 (IoU=0.5)
on Charades-STA, compared to state-of-the-art video large language models,
holding the potential to serve as a versatile video assistant for long-form
video comprehension tasks and satisfy realistic user requirements.

该研究提出了 TimeChat，一种针对长视频理解的时态敏感多模态大型语言模型。该模型通过两个关键的架构贡献实现：1) 能够将每帧的视觉内容与时间戳绑定的时间感知帧编码器，和 2) 一种产生适应不同持续时间视频的长度可变视频令牌序列的滑动视频 Q-Former。此外，我们构建了一个调整指令的数据集，包括 6 个任务和总共 12.5 万个实例，以进一步提高 TimeChat 的指令遵循性能。在各种视频理解任务上的实验结果，如密集字幕生成、时间定位和重点检测，展示了 TimeChat 强大的零样本时态定位和推理能力。例如，在 YouCook2 上，它在 F1 评分上提升了 9.2，在 CIDEr 上提升了 2.8，在 QVHighlights 上的 HIT@1 提升了 5.8，在 Charades-STA 上的 R@1 (IoU=0.5) 提升了 27.5，与业界领先的视频大型语言模型相比，具备作为长视频理解任务的通用视频助手并满足真实用户需求的潜力。