Vision-Language models (VLMs) that use contrastive language-image
pre-training have shown promising zero-shot classification performance.
However, their performance on imbalanced dataset is relatively poor, where the
distribution of classes in the training dataset is skewed, leading to poor
performance in predicting minority classes. For instance, CLIP achieved only 5%
accuracy on the iNaturalist18 dataset. We propose to add a lightweight decoder
to VLMs to avoid OOM (out of memory) problem caused by large number of classes
and capture nuanced features for tail classes. Then, we explore improvements of
VLMs using prompt tuning, fine-tuning, and incorporating imbalanced algorithms
such as Focal Loss, Balanced SoftMax and Distribution Alignment. Experiments
demonstrate that the performance of VLMs can be further boosted when used with
decoder and imbalanced methods. Specifically, our improved VLMs significantly
outperforms zero-shot classification by an average accuracy of 6.58%, 69.82%,
and 6.17%, on ImageNet-LT, iNaturalist18, and Places-LT, respectively. We
further analyze the influence of pre-training data size, backbones, and
training cost. Our study highlights the significance of imbalanced learning
algorithms in face of VLMs pre-trained by huge data. We release our code at
this https URL

本文针对 Vision-Language models 在处理 imbalanced dataset 时性能较差的问题，提出了加入 lightweight decoder 和 imbalanced 方法的改进方案，并在 ImageNet-LT iNaturalist18 和 Places-LT 三个数据集上进行了实验，证明改进后的 VLMs 相较于原来的 zero-shot classification 方法，在准确率上有显著提升。