Vision language models (VLMs) perceive the world through a combination of a visual encoder and a large language model (LLM). The visual encoder, pre-trained on large-scale vision-text datasets, provides zero-shot generalization to visual data, and the LLM endows its high reasoning ability to VLMs. It leads VLMs to achieve high performance on wide benchmarks without fine-tuning, exhibiting zero or few-shot capability. However, recent studies show that VLMs are vulnerable to hallucination. This undesirable behavior degrades reliability and credibility, thereby making users unable to fully trust the output from VLMs. To enhance trustworthiness and better tackle the hallucination of VLMs, we curate a new evaluation dataset, called the BEfore-AFter hallucination dataset (BEAF), and introduce new metrics: True Understanding (TU), IGnorance (IG), StuBbornness (SB), and InDecision (ID). Unlike prior works that focus only on constructing questions and answers, the key idea of our benchmark is to manipulate visual scene information by image editing models and to design the metrics based on scene changes. This allows us to clearly assess whether VLMs correctly understand a given scene by observing the ability to perceive changes. We also visualize image-wise object relationship by virtue of our two-axis view: vision and text. Upon evaluating VLMs with our dataset, we observed that our metrics reveal different aspects of VLM hallucination that have not been reported before. Project page: \url{https://beafbench.github.io/}

为了提高视觉语言模型（VLMs）的可信度并更好地解决其幻觉问题，我们创建了一个新的评估数据集，称为BEfore-AFter幻觉数据集（BEAF），并引入了新的指标：True Understanding (TU)，IGnorance (IG)，StuBbornness (SB)，和 InDecision (ID)。通过图像编辑模型操纵视觉场景信息，并根据场景变化设计指标，我们可以清楚地评估VLMs是否能正确理解给定的场景，以观察其感知能力。

BEAF：评估视觉语言模型中的幻觉的前后变化