Recent advances in instruction-tuned Large Vision-Language Models (LVLMs)
have imbued the models with the ability to generate high-level, image-grounded
explanations with ease. While such capability is largely attributed to the rich
world knowledge contained within the Large Language Models (LLMs), our work
reveals their shortcomings in fine-grained visual categorization (FGVC) across
six different benchmark settings. Most recent state-of-the-art LVLMs like
LLaVa-1.5, InstructBLIP and GPT-4V not only severely deteriorate in terms of
classification performance, e.g., average drop of 65.58 in EM for Stanford Dogs
for LLaVA-1.5, but also struggle to generate an accurate explanation with
detailed attributes based on the concept that appears within an input image
despite their capability to generate holistic image-level descriptions.
In-depth analyses show that instruction-tuned LVLMs exhibit modality gap,
showing discrepancy when given textual and visual inputs that correspond to the
same concept, preventing the image modality from leveraging the rich parametric
knowledge within the LLMs. In an effort to further the community's endeavor in
this direction, we propose a multiple granularity attribute-centric evaluation
benchmark, Finer, which aims to establish a ground to evaluate LVLMs'
fine-grained visual comprehension ability and provide significantly improved
explainability.

最近在指导的大型视觉语言模型方面取得的进展，使得模型能够轻松生成高层次的基于图像的解释。然而，我们的工作揭示了这些模型在细粒度的视觉分类方面的缺陷，并且我们提出了一个多粒度属性为中心的评估基准，用于评估大型视觉语言模型的细粒度视觉理解能力并提供显著改进的可解释性。