This paper explores preference distillation for large vision language models
(LVLMs), improving their ability to generate helpful and faithful responses
anchoring the visual context. We first build a vision-language feedback
(VLFeedback) dataset utilizing AI annotation. Specifically, responses are
generated by models sampled from 12 LVLMs, conditioned on multi-modal
instructions sourced from various datasets. We adopt GPT-4V to assess the
generated outputs regarding helpfulness, visual faithfulness, and ethical
considerations. Furthermore, the preference supervision is distilled into
Qwen-VL-Chat through the direct preference optimization (DPO) method. The
resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME
benchmark regarding the perception and cognition capabilities, respectively.
Silkie also demonstrates reduced hallucination by setting a new
state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis
shows that DPO with our VLFeedback dataset mainly boosts the fine-grained
perception and complex cognition abilities of LVLMs, leading to more
comprehensive improvements compared to human-annotated preference datasets.

本研究探讨大型视觉语言模型 (LVLMs) 的偏好提取，提高其生成有助于和忠实于视觉上下文的回应能力。通过构建视觉语言反馈 (VLFeedback) 数据集并采用 Direct Preference Optimization（DPO）方法，研究人员实现了 Silkie 模型，其在感知和认知能力方面相对提高了 6.9％和 9.5％，同时在 MMHal-Bench 基准测试中取得了 3.02 的最优成绩，证明 DPO 与 VLFeedback 数据集主要提升了 LVLMs 的细粒度感知和复杂认知能力，相比人工标注的偏好数据集实现了更全面的改进。