Large language models (LLMs) has been effectively used for many computer
vision tasks, including image classification. In this paper, we present a
simple yet effective approach for zero-shot image classification using
multimodal LLMs. By employing multimodal LLMs, we generate comprehensive
textual representations from input images. These textual representations are
then utilized to generate fixed-dimensional features in a cross-modal embedding
space. Subsequently, these features are fused together to perform zero-shot
classification using a linear classifier. Our method does not require prompt
engineering for each dataset; instead, we use a single, straightforward, set of
prompts across all datasets. We evaluated our method on several datasets, and
our results demonstrate its remarkable effectiveness, surpassing benchmark
accuracy on multiple datasets. On average over ten benchmarks, our method
achieved an accuracy gain of 4.1 percentage points, with an increase of 6.8
percentage points on the ImageNet dataset, compared to prior methods. Our
findings highlight the potential of multimodal LLMs to enhance computer vision
tasks such as zero-shot image classification, offering a significant
improvement over traditional methods.

使用多模态大语言模型 (Multimodal LLMs) 的简单且有效方法实现了零样本图像分类，通过生成全面的文本表示从而在交叉模态嵌入空间中生成固定维度特征，在线性分类器上融合这些特征以进行分类，取得了令人瞩目的效果。