Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images. While current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities, this work systematically investigates multi-object hallucination, examining how models misperceive (e.g., invent nonexistent objects or become distracted) when tasked with focusing on multiple objects simultaneously. We introduce Recognition-based Object Probing Evaluation (ROPE), an automated evaluation protocol that considers the distribution of object classes within a single image during testing and uses visual referring prompts to eliminate ambiguity. With comprehensive empirical studies and analysis of potential factors leading to multi-object hallucination, we found that (1) LVLMs suffer more hallucinations when focusing on multiple objects compared to a single object. (2) The tested object class distribution affects hallucination behaviors, indicating that LVLMs may follow shortcuts and spurious correlations.(3) Hallucinatory behaviors are influenced by data-specific factors, salience and frequency, and model intrinsic behaviors. We hope to enable LVLMs to recognize and reason about multiple objects that often occur in realistic visual scenes, provide insights, and quantify our progress towards mitigating the issues.

大规模视觉语言模型（LVLMs）在对象幻觉方面经常遇到困难，会产生图像中不存在的对象。本研究系统地调查了多对象幻觉，在同时关注多个对象任务时，模型如何错误地理解（如发明不存在的对象或分心）。我们引入了基于识别的对象探针评估（ROPE），这是一种自动化评估协议，在测试过程中考虑到单个图像中的对象类别分布，并使用视觉引用提示来消除歧义。通过全面的实证研究和分析导致多对象幻觉的潜在因素，我们发现（1）与关注单个对象相比，LVLMs在关注多个对象时更容易产生幻觉。（2）测试的对象类别分布会影响幻觉行为，表明LVLMs可能会遵循捷径和虚假相关性。（3）幻觉行为受到数据特定因素、显著性和频率以及模型内在行为的影响。我们希望使LVLMs能够识别和推理常见于现实视觉场景中的多个对象，提供见解并量化我们在解决这些问题上的进展。

视觉语言模型中的多物体幻觉