Recently, several multi-modal models have been developed for joint image and
language understanding, which have demonstrated impressive chat abilities by
utilizing advanced large language models (LLMs). The process of developing such
models is straightforward yet effective. It involves pre-training an adaptation
module to align the semantics of the vision encoder and language model,
followed by fine-tuning on the instruction-following data. However, despite the
success of this pipeline in image and language understanding, its effectiveness
in joint video and language understanding has not been widely explored. In this
paper, we aim to develop a novel multi-modal foundation model capable of
perceiving video, image, and language within a general framework. To achieve
this goal, we introduce Valley: Video Assistant with Large Language model
Enhanced ability. Specifically, our proposed Valley model is designed with a
simple projection module that bridges video, image, and language modalities,
and is further unified with a multi-lingual LLM. We also collect multi-source
vision-text pairs and adopt a spatio-temporal pooling strategy to obtain a
unified vision encoding of video and image input for pre-training. Furthermore,
we generate multi-task instruction-following video data, including multi-shot
captions, long video descriptions, action recognition, causal relationship
inference, etc. To obtain the instruction-following data, we design diverse
rounds of task-oriented conversations between humans and videos, facilitated by
ChatGPT. Qualitative examples demonstrate that our proposed model has the
potential to function as a highly effective multilingual video assistant that
can make complex video understanding scenarios easy. Code, data, and models
will be available at this https URL

本文旨在开发一个新颖的多模态基础模型 ——Valley: Video Assistant with Large Language model Enhanced ability，实现对视频、图像和语言的感知，并通过 ChatGPT 设计的多任务指令追随视频数据，在人类和视频之间进行各种任务定向的对话，为多语言的视频助手提供高度有效的支持。