Multi-modal Large Language Models (MLLMs) have demonstrated their ability to
perceive objects in still images, but their application in video-related tasks,
such as object tracking, remains understudied. This lack of exploration is
primarily due to two key challenges. Firstly, extensive pretraining on
large-scale video datasets is required to equip MLLMs with the capability to
perceive objects across multiple frames and understand inter-frame
relationships. Secondly, processing a large number of frames within the context
window of Large Language Models (LLMs) can impose a significant computational
burden. To address the first challenge, we introduce ElysiumTrack-1M, a
large-scale video dataset paired with novel tasks: Referring Single Object
Tracking (RSOT) and Video Referring Expression Generation (Video-REG).
ElysiumTrack-1M contains 1.27 million annotated video frames with corresponding
object boxes and descriptions. Leveraging this dataset, we conduct training of
MLLMs and propose a token-compression model T-Selector to tackle the second
challenge. Our proposed approach, Elysium: Exploring Object-level Perception in
Videos via MLLM, is an end-to-end trainable MLLM that makes the first attempt
to conduct object-level tasks in videos without requiring any additional
plug-in or expert models.

通过在大型视频数据集上进行大规模预训练，我们提出了一种全新的多模态大型语言模型（MLLM），名为 Elysium，该模型可以在视频中进行物体级任务，而无需任何其他插件或专家模型。