Multimodal Large Language Models (MLLMs) have recently demonstrated
impressive capabilities in multimodal understanding, reasoning, and
interaction. However, existing MLLMs prevalently suffer from serious
hallucination problems, generating text that is not factually grounded in
associated images. The problem makes existing MLLMs untrustworthy and thus
impractical in real-world (especially high-stakes) applications. To address the
challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior
alignment from fine-grained correctional human feedback. Specifically, RLHF-V
collects human preference in the form of segment-level corrections on
hallucinations, and performs dense direct preference optimization over the
human feedback. Comprehensive experiments on five benchmarks in both automatic
and human evaluation show that, RLHF-V can enable substantially more
trustworthy MLLM behaviors with promising data and computation efficiency.
Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the
hallucination rate of the base MLLM by 34.8%, outperforming the concurrent
LLaVA-RLHF trained on 10k annotated data. The final model achieves
state-of-the-art performance in trustworthiness among open-source MLLMs, and
shows better robustness than GPT-4V in preventing hallucinations aroused from
over-generalization. We open-source our code, model, and data at
this https URL

通过细粒度纠正的人类反馈，RLHF-V 通过行为对齐增强了 Multi-Modal Large Language Models (MLLMs) 的可靠性。通过在自动化和人类评估中进行全面实验，RLHF-V 能够以有希望的数据和计算效率，在五个基准测试中显著降低 MLLM 的错觉率，并达到开放源代码 MLLM 中可靠性方面的最新技术水平。