This paper does not present a novel method. Instead, it delves into an
essential, yet must-know baseline in light of the latest advancements in
Generative Artificial Intelligence (GenAI): the utilization of GPT-4 for visual
understanding. Our study centers on the evaluation of GPT-4's linguistic and
visual capabilities in zero-shot visual recognition tasks. Specifically, we
explore the potential of its generated rich textual descriptions across various
categories to enhance recognition performance without any training.
Additionally, we evaluate its visual proficiency in directly recognizing
diverse visual content. To achieve this, we conduct an extensive series of
experiments, systematically quantifying the performance of GPT-4 across three
modalities: images, videos, and point clouds. This comprehensive evaluation
encompasses a total of 16 widely recognized benchmark datasets, providing top-1
and top-5 accuracy metrics. Our study reveals that leveraging GPT-4's advanced
linguistic knowledge to generate rich descriptions markedly improves zero-shot
recognition. In terms of visual proficiency, GPT-4V's average performance
across 16 datasets sits roughly between the capabilities of OpenAI-CLIP's ViT-L
and EVA-CLIP's ViT-E. We hope that this research will contribute valuable data
points and experience for future studies. We release our code at
this https URL

本研究对 GPT-4 在零样本视觉识别任务中的语言和视觉能力进行了评估，发现利用 GPT-4 生成丰富的文本描述明显提高了零样本识别性能，并且在视觉熟练度方面，GPT-4V 在 16 个基准数据集中表现介于 OpenAI-CLIP 的 ViT-L 和 EVA-CLIP 的 ViT-E 之间。