The emergence of multimodal large models (MLMs) has significantly advanced
the field of visual understanding, offering remarkable capabilities in the
realm of visual question answering (VQA). Yet, the true challenge lies in the
domain of knowledge-intensive VQA tasks, which necessitate not just recognition
of visual elements, but also a deep comprehension of the visual information in
conjunction with a vast repository of learned knowledge. To uncover such
capabilities of MLMs, particularly the newly introduced GPT-4V, we provide an
in-depth evaluation from three perspectives: 1) Commonsense Knowledge, which
assesses how well models can understand visual cues and connect to general
knowledge; 2) Fine-grained World Knowledge, which tests the model's skill in
reasoning out specific knowledge from images, showcasing their proficiency
across various specialized fields; 3) Comprehensive Knowledge with
Decision-making Rationales, which examines model's capability to provide
logical explanations for its inference, facilitating a deeper analysis from the
interpretability perspective. Extensive experiments indicate that GPT-4V
achieves SOTA performance on above three tasks. Interestingly, we find that: a)
GPT-4V demonstrates enhanced reasoning and explanation when using composite
images as few-shot; b) GPT-4V produces severe hallucinations when dealing with
world knowledge, highlighting the future need for advancements in this research
direction.

通过从三个角度对 GPT-4V 进行深入评估，即常识知识、细粒度世界知识和具有决策理由的综合知识，我们发现 GPT-4V 在这三项任务上均取得了最好的性能，并且在使用复合图像进行少样本学习时具有增强的推理和解释能力，但在处理世界知识时可能导致严重的错觉，未来仍需要在这个研究方向上进行改进。