The troubling rise of hallucination presents perhaps the most significant
impediment to the advancement of responsible AI. In recent times, considerable
research has focused on detecting and mitigating hallucination in Large
Language Models (LLMs). However, it's worth noting that hallucination is also
quite prevalent in Vision-Language models (VLMs). In this paper, we offer a
fine-grained discourse on profiling VLM hallucination based on two tasks: i)
image captioning, and ii) Visual Question Answering (VQA). We delineate eight
fine-grained orientations of visual hallucination: i) Contextual Guessing, ii)
Identity Incongruity, iii) Geographical Erratum, iv) Visual Illusion, v) Gender
Anomaly, vi) VLM as Classifier, vii) Wrong Reading, and viii) Numeric
Discrepancy. We curate Visual HallucInation eLiciTation (VHILT), a publicly
available dataset comprising 2,000 samples generated using eight VLMs across
two tasks of captioning and VQA along with human annotations for the categories
as mentioned earlier.

本研究针对视觉 - 语言模型中的幻觉进行细致的分析，并通过图像字幕和视觉问答两个任务，确定了八个精细化的视觉幻觉方向：上下文猜测、身份不一致、地理错误、视觉错觉、性别异常、VLM 作为分类器、错误阅读和数字不一致。同时，还提供了一个包含 2,000 个样本的公开数据集 VHILT，用于研究这些类别的视觉幻觉。