Large language models (LLMs) have emerged as powerful general-purpose
interfaces for many machine learning problems. Recent work has adapted LLMs to
generative visual tasks like image captioning, visual question answering, and
visual chat, using a relatively small amount of instruction-tuning data. In
this paper, we explore whether modern LLMs can also be adapted to classifying
an image into a set of categories. First, we evaluate multimodal LLMs that are
tuned for generative tasks on zero-shot image classification and find that
their performance is far below that of specialized models like CLIP. We then
propose an approach for light fine-tuning of LLMs using the same contrastive
image-caption matching objective as CLIP. Our results show that LLMs can,
indeed, achieve good image classification performance when adapted this way.
Our approach beats state-of-the-art mLLMs by 13% and slightly outperforms
contrastive learning with a custom text model, while also retaining the LLM's
generative abilities. LLM initialization appears to particularly help
classification in domains under-represented in the visual pre-training data.

研究论文简介：本文研究了大型语言模型在图像分类方面的应用，通过对多模态语言模型进行轻微微调，使用对比式图像 - 标题匹配目标，取得了比目前最先进的 MLLMs 提高了 13% 的图像分类性能，同时保留了语言模型的生成能力。