Recent advances in the development of vision-language models (VLMs) are
yielding remarkable success in recognizing visual semantic content, including
impressive instances of compositional image understanding. Here, we introduce
the novel task of \textit{Visual Data-Type Identification}, a basic perceptual
skill with implications for data curation (e.g., noisy data-removal from large
datasets, domain-specific retrieval) and autonomous vision (e.g.,
distinguishing changing weather conditions from camera lens staining). We
develop two datasets consisting of animal images altered across a diverse set
of 27 visual \textit{data-types}, spanning four broad categories. An extensive
zero-shot evaluation of 39 VLMs, ranging from 100M to 80B parameters, shows a
nuanced performance landscape. While VLMs are reasonably good at identifying
certain stylistic \textit{data-types}, such as cartoons and sketches, they
struggle with simpler \textit{data-types} arising from basic manipulations like
image rotations or additive noise. Our findings reveal that (i) model scaling
alone yields marginal gains for contrastively-trained models like CLIP, and
(ii) there is a pronounced drop in performance for the largest
auto-regressively trained VLMs like OpenFlamingo. This finding points to a
blind spot in current frontier VLMs: they excel in recognizing semantic content
but fail to acquire an understanding of visual \textit{data-types} through
scaling. By analyzing the pre-training distributions of these models and
incorporating \textit{data-type} information into the captions during
fine-tuning, we achieve a significant enhancement in performance. By exploring
this previously uncharted task, we aim to set the stage for further advancing
VLMs to equip them with visual data-type understanding. Code and datasets are
released \href{this https URL}{here}.

最新的视觉语言模型（VLM）在识别视觉语义内容方面取得了显著的成功，但在识别视觉数据类型方面存在盲区。本文通过引入 “视觉数据类型识别” 这一新任务，探索并提升 VLM 在识别视觉数据类型方面的能力，通过分析模型的预训练分布，并在微调过程中引入数据类型信息，实现了显著的性能提升。