This paper undertakes an empirical study to revisit the latest advancements
in Multimodal Large Language Models (MLLMs): Video Assistant. This study,
namely FreeVA, aims to extend existing image-based MLLM to the video domain in
a training-free manner. The study provides an essential, yet must-know
baseline, and reveals several surprising findings: 1) FreeVA, leveraging only
offline image-based MLLM without additional training, excels in zero-shot video
question-answering (e.g., MSVD-QA, ActivityNet-QA, and MSRVTT-QA), even
surpassing state-of-the-art methods that involve video instruction tuning. 2)
While mainstream video-based MLLMs typically initialize with an image-based
MLLM (e.g., LLaVA) and then fine-tune using video instruction tuning, the study
indicates that utilizing the widely adopted VideoInstruct-100K for video
instruction tuning doesn't actually lead to better performance compared to not
training at all. 3) The commonly used evaluation metrics in existing works are
significantly influenced by changes in the GPT API version over time. If
ignored, this could affect the fairness and uniformity of comparisons between
different methods and impact the analysis and judgment of researchers in the
field. The advancement of MLLMs is currently thriving, drawing numerous
researchers into the field. We aim for this work to serve as a plug-and-play,
simple yet effective baseline, encouraging the direct evaluation of existing
MLLMs in video domain while also standardizing the field of video
conversational models to a certain extent. Also, we encourage researchers to
reconsider: Have current video MLLM methods truly acquired knowledge beyond
image MLLM? Code is available at this https URL

通过对多模态大型语言模型（MLLMs）的最新进展进行实证研究，本文介绍了一个名为 FreeVA 的研究，旨在以无需训练的方式将现有基于图像的 MLLM 扩展到视频领域。研究发现，仅利用离线图像 MLLM 而无需额外训练的 FreeVA，在零样本视频问答（例如 MSVD-QA、ActivityNet-QA 和 MSRVTT-QA）方面表现出色，甚至超过了涉及视频指令调整的最先进方法。此外，本文还指出，与根本不训练相比，利用广泛采用的 VideoInstruct-100K 进行视频指令调整并不会导致更好的性能。最后，本文强调评估指标在不同方式的比较中受 GPT API 版本变化的显著影响，如果忽略这一点，可能会影响研究者的分析和判断。

FreeVA: 离线 MLLM 作为无需训练的视频助手

FreeVA: Offline MLLM as Training-Free Video Assistant

Recently, several multi-modal models have been developed for joint image and
language understanding, which have demonstrated impressive chat abilities by
utilizing advanced large language models (LLMs). The process of developing such
models is straightforward yet effective. It involves pre-training an adaptation
module to align the semantics of the vision encoder and language model,
followed by fine-tuning on the instruction-following data. However, despite the
success of this pipeline in image and language understanding, its effectiveness
in joint video and language understanding has not been widely explored. In this
paper, we aim to develop a novel multi-modal foundation model capable of
perceiving video, image, and language within a general framework. To achieve
this goal, we introduce Valley: Video Assistant with Large Language model
Enhanced ability. Specifically, our proposed Valley model is designed with a
simple projection module that bridges video, image, and language modalities,
and is further unified with a multi-lingual LLM. We also collect multi-source
vision-text pairs and adopt a spatio-temporal pooling strategy to obtain a
unified vision encoding of video and image input for pre-training. Furthermore,
we generate multi-task instruction-following video data, including multi-shot
captions, long video descriptions, action recognition, causal relationship
inference, etc. To obtain the instruction-following data, we design diverse
rounds of task-oriented conversations between humans and videos, facilitated by
ChatGPT. Qualitative examples demonstrate that our proposed model has the
potential to function as a highly effective multilingual video assistant that
can make complex video understanding scenarios easy. Code, data, and models
will be available at this https URL

本文旨在开发一个新颖的多模态基础模型 ——Valley: Video Assistant with Large Language model Enhanced ability，实现对视频、图像和语言的感知，并通过 ChatGPT 设计的多任务指令追随视频数据，在人类和视频之间进行各种任务定向的对话，为多语言的视频助手提供高度有效的支持。