The advent of Vision Language Models (VLM) has allowed researchers to
investigate the visual understanding of a neural network using natural
language. Beyond object classification and detection, VLMs are capable of
visual comprehension and common-sense reasoning. This naturally led to the
question: How do VLMs respond when the image itself is inherently unreasonable?
To this end, we present IllusionVQA: a diverse dataset of challenging optical
illusions and hard-to-interpret scenes to test the capability of VLMs in two
distinct multiple-choice VQA tasks - comprehension and soft localization.
GPT4V, the best-performing VLM, achieves 62.99% accuracy (4-shot) on the
comprehension task and 49.7% on the localization task (4-shot and
Chain-of-Thought). Human evaluation reveals that humans achieve 91.03% and 100%
accuracy in comprehension and localization. We discover that In-Context
Learning (ICL) and Chain-of-Thought reasoning substantially degrade the
performance of GeminiPro on the localization task. Tangentially, we discover a
potential weakness in the ICL capabilities of VLMs: they fail to locate optical
illusions even when the correct answer is in the context window as a few-shot
example.

Vision Language Models are tested on the IllusionVQA dataset, revealing their performance and weaknesses in comprehension and soft localization tasks, particularly in the context of optical illusions and In-Context Learning.