Image classification is one of the most fundamental capabilities of machine
vision intelligence. In this work, we revisit the image classification task
using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We
find that existing proprietary and public VLMs, despite often using CLIP as a
vision encoder and having many more parameters, significantly underperform CLIP
on standard image classification benchmarks like ImageNet. To understand the
reason, we explore several hypotheses concerning the inference algorithms,
training objectives, and data processing in VLMs. Our analysis reveals that the
primary cause is data-related: critical information for image classification is
encoded in the VLM's latent space but can only be effectively decoded with
enough training data. Specifically, there is a strong correlation between the
frequency of class exposure during VLM training and instruction-tuning and the
VLM's performance in those classes; when trained with sufficient data, VLMs can
match the accuracy of state-of-the-art classification models. Based on these
findings, we enhance a VLM by integrating classification-focused datasets into
its training, and demonstrate that the enhanced classification performance of
the VLM transfers to its general capabilities, resulting in an improvement of
11.8% on the newly collected ImageWikiQA dataset.

基于本文的研究发现，将分类聚焦的数据集整合到可视语言模型的训练中可以提高其分类性能，并将性能提升转移到其一般能力上，对于新收集的 ImageWikiQA 数据集，准确率提高了 11.8%。