Enabling large language models (LLMs) to read videos is vital for multimodal
LLMs. Existing works show promise on short videos whereas long video (longer
than e.g.~1 minute) comprehension remains challenging. The major problem lies
in the over-compression of videos, i.e., the encoded video representations are
not enough to represent the whole video. To address this issue, we propose Long
Video Chat (LVChat), where Frame-Scalable Encoding (FSE) is introduced to
dynamically adjust the number of embeddings in alignment with the duration of
the video to ensure long videos are not overly compressed into a few
embeddings. To deal with long videos whose length is beyond videos seen during
training, we propose Interleaved Frame Encoding (IFE), repeating positional
embedding and interleaving multiple groups of videos to enable long video
input, avoiding performance degradation due to overly long videos. Experimental
results show that LVChat significantly outperforms existing methods by up to
27\% in accuracy on long-video QA datasets and long-video captioning
benchmarks. Our code is published at this https URL

通过引入动态调整嵌入数量以适应视频时长的方式，以及重复位置嵌入和交错多组视频以处理超长视频输入的方法，我们提出了 Long Video Chat (LVChat)，在长视频问答数据集和长视频字幕基准测试中，实验证明 LVChat 在准确度上优于现有方法高达 27%。