Multimodal Large Language Models (MLLMs) have demonstrated profound
capabilities in understanding multimodal information, covering from Image LLMs
to the more complex Video LLMs. Numerous studies have illustrated their
exceptional cross-modal comprehension. Recently, integrating video foundation
models with large language models to build a comprehensive video understanding
system has been proposed to overcome the limitations of specific pre-defined
vision tasks. However, the current advancements in Video LLMs tend to overlook
the foundational contributions of Image LLMs, often opting for more complicated
structures and a wide variety of multimodal data for pre-training. This
approach significantly increases the costs associated with these methods.In
response to these challenges, this work introduces an efficient method that
strategically leverages the priors of Image LLMs, facilitating a
resource-efficient transition from Image to Video LLMs. We propose RED-VILLM, a
Resource-Efficient Development pipeline for Video LLMs from Image LLMs, which
utilizes a temporal adaptation plug-and-play structure within the image fusion
module of Image LLMs. This adaptation extends their understanding capabilities
to include temporal information, enabling the development of Video LLMs that
not only surpass baseline performances but also do so with minimal
instructional data and training resources. Our approach highlights the
potential for a more cost-effective and scalable advancement in multimodal
models, effectively building upon the foundational work of Image LLMs.

提出一种高效的方法，通过调整图像多模态语言模型的图像融合模块，利用图像多模态语言模型的先验知识，实现从图像到视频多模态语言模型的资源高效转换，从而提高视频多模态语言模型的时间理解能力，以更少的数据和资源进行训练。