Recent advancements in large language models have influenced the development
of video large multimodal models (VLMMs). The previous approaches for VLMMs
involved Supervised Fine-Tuning (SFT) with instruction-tuned datasets,
integrating LLM with visual encoders, and adding additional learnable modules.
Video and text multimodal alignment remains challenging, primarily due to the
deficient volume and quality of multimodal instruction-tune data compared to
text-only data. We present a novel alignment strategy that employs multimodal
AI system to oversee itself called Reinforcement Learning from AI Feedback
(RLAIF), providing self-preference feedback to refine itself and facilitating
the alignment of video and text modalities. In specific, we propose
context-aware reward modeling by providing detailed video descriptions as
context during the generation of preference feedback in order to enrich the
understanding of video content. Demonstrating enhanced performance across
diverse video benchmarks, our multimodal RLAIF approach, VLM-RLAIF, outperforms
existing approaches, including the SFT model. We commit to open-sourcing our
code, models, and datasets to foster further research in this area.

大型语言模型对视频大型多模型的发展产生了影响。我们提出了一种新的多模态智能系统对齐策略，称为从 AI 反馈中的强化学习，通过提供详细的视频描述来丰富视频内容的理解，以改进视频和文本内容的对齐效果。我们的方法 VLM-RLAIF 在多种视频基准测试中表现出卓越的性能，超过了现有的方法，包括 SFT 模型。我们致力于开源我们的代码、模型和数据集，以促进该领域的进一步研究。

利用强化学习从 AI 反馈中优化大型多模态视频模型

Tuning Large Multimodal Models for Videos using Reinforcement Learning  from AI Feedback

This work aims to create a multimodal AI system that chats with humans and
shares relevant photos. While earlier works were limited to dialogues about
specific objects or scenes within images, recent works have incorporated images
into open-domain dialogues. However, their response generators are unimodal,
accepting text input but no image input, thus prone to generating responses
contradictory to the images shared in the dialogue. Therefore, this work
proposes a complete chatbot system using two multimodal deep learning models:
an image retriever that understands texts and a response generator that
understands images. The image retriever, implemented by ViT and BERT, selects
the most relevant image given the dialogue history and a database of images.
The response generator, implemented by ViT and GPT-2/DialoGPT, generates an
appropriate response given the dialogue history and the most recently retrieved
image. The two models are trained and evaluated on PhotoChat, an open-domain
dialogue dataset in which a photo is shared in each session. In automatic
evaluation, the proposed image retriever outperforms existing baselines VSE++
and SCAN with Recall@1/5/10 of 0.1/0.3/0.4 and MRR of 0.2 when ranking 1,000
images. The proposed response generator also surpasses the baseline Divter with
PPL of 16.9, BLEU-1/2 of 0.13/0.03, and Distinct-1/2 of 0.97/0.86, showing a
significant improvement in PPL by -42.8 and BLEU-1/2 by +0.07/0.02. In human
evaluation with a Likert scale of 1-5, the complete multimodal chatbot system
receives higher image-groundedness of 4.3 and engagingness of 4.3, along with
competitive fluency of 4.1, coherence of 3.9, and humanness of 3.1, when
compared to other chatbot variants. The source code is available at:
this https URL

该研究旨在创建一个多模态人工智能系统，可以与人类交谈并分享相关图片，其中包括了图片检索模块和图片理解生成回复的模块。使用 ViT，BERT，GPT-2/DialoGPT 三个模型，在自动生成的和人工评估中结果显示，在开放式场景的图像识别中表现优异。