Amidst the advancements in image-based Large Vision-Language Models
(image-LVLM), the transition to video-based models (video-LVLM) is hindered by
the limited availability of quality video data. This paper addresses the
challenge by leveraging the visual commonalities between images and videos to
efficiently evolve image-LVLMs into video-LVLMs. We present a cost-effective
video-LVLM that enhances model architecture, introduces innovative training
strategies, and identifies the most effective types of video instruction data.
Our innovative weighted token sampler significantly compresses the visual token
numbers of each video frame, effectively cutting computational expenses. We
also find that judiciously using just 10% of the video data, compared to prior
video-LVLMs, yields impressive results during various training phases.
Moreover, we delve into the influence of video instruction data in
limited-resource settings, highlighting the significance of incorporating video
training data that emphasizes temporal understanding to enhance model
performance. The resulting Fewer Tokens and Fewer Videos LVLM (FTFV-LVLM)
exhibits exceptional performance across video and image benchmarks, validating
our model's design and training approaches.

通过利用图像和视频之间的视觉相似性，本文介绍了一种成本效益较高的视频 - LVLM 模型，通过改进模型结构、引入创新的训练策略，并确定最有效的视频指令数据类型，实现了将基于图像的 LVLM 模型高效演化为视频 - LVLM 模型，并在有限资源环境下强调了时间理解的视频培训数据的重要性，提高了模型性能。