This paper proposes Video-Teller, a video-language foundation model that
leverages multi-modal fusion and fine-grained modality alignment to
significantly enhance the video-to-text generation task. Video-Teller boosts
the training efficiency by utilizing frozen pretrained vision and language
modules. It capitalizes on the robust linguistic capabilities of large language
models, enabling the generation of both concise and elaborate video
descriptions. To effectively integrate visual and auditory information,
Video-Teller builds upon the image-based BLIP-2 model and introduces a cascaded
Q-Former which fuses information across frames and ASR texts. To better guide
video summarization, we introduce a fine-grained modality alignment objective,
where the cascaded Q-Former's output embedding is trained to align with the
caption/summary embedding created by a pretrained text auto-encoder.
Experimental results demonstrate the efficacy of our proposed video-language
foundation model in accurately comprehending videos and generating coherent and
precise language descriptions. It is worth noting that the fine-grained
alignment enhances the model's capabilities (4% improvement of CIDEr score on
MSR-VTT) with only 13% extra parameters in training and zero additional cost in
inference.

该研究提出了一种名为 Video-Teller 的视频语言基础模型，通过多模态融合和细粒度模态对齐来显著增强视频到文本生成任务。实验结果表明，该模型在准确理解视频和生成连贯且精确的语言描述方面取得了显著效果。