We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in addition to the conventional visual tokenizer, which maps a sequence of tokens over multiple frames into a compact set of visual tokens. This enables BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32 vs. 4608 tokens). We explore different types of temporal encoders, including learnable spatio-temporal pooling as well as sequential models like Token Turing Machines. We experimentally confirm that BLIP-3-Video obtains video question-answering accuracies comparable to much larger state-of-the-art models (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using fewer visual tokens. The project website is at https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html

该研究针对视频处理中的效率问题，提出了一种新的多模态语言模型BLIP-3-Video，利用时间编码器显著减少了对视觉标记的需求。实验结果表明，BLIP-3-Video在视频问答准确性上与更大规模的先进模型相当，但模型体积更小、效率更高。

xGen-MM-Vid (BLIP-3-Video): 只需32个标记即可表示视频