Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs, undermining their reliability. A comprehensive quantitative evaluation is necessary to identify and understand the extent of hallucinations in these models. However, existing benchmarks are often limited in scope, focusing mainly on object hallucinations. Furthermore, current evaluation methods struggle to effectively address the subtle semantic distinctions between model outputs and reference data, as well as the balance between hallucination and informativeness. To address these issues, we introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases. Moreover, we propose an large language model (LLM)-based two-stage evaluation framework that generalizes the popular CHAIR metric and incorporates both faithfulness and coverage into the evaluation. Experiments on 10 established LVLMs demonstrate that our evaluation metric is more comprehensive and better correlated with humans than existing work when evaluating on our challenging human annotated benchmark dataset. Our work also highlights the critical balance between faithfulness and coverage of model outputs, and encourages future works to address hallucinations in LVLMs while keeping their outputs informative.

大型视觉语言模型（LVLMs）在生成看似可信但事实上不正确的输出时存在幻觉问题，这损害了它们的可靠性。为了解决这些问题，我们介绍了一个多维度的评估基准和基于大型语言模型（LLM）的两阶段评估框架，通过对现有LVLMs进行实验，证明了我们的评估方法比现有工作更全面、与人类更相关，并且强调了在保持输出信息性的同时解决LVLMs中的幻觉问题的关键平衡。

VALOR-EVAL: 大型视觉语言模型的整体覆盖和忠实度评估