Contrastive Language-Image Pretraining (CLIP) stands out as a prominent
method for image representation learning. Various architectures, from vision
transformers (ViTs) to convolutional networks (ResNets) have been trained with
CLIP to serve as general solutions to diverse vision tasks. This paper explores
the differences across various CLIP-trained vision backbones. Despite using the
same data and training objective, we find that these architectures have notably
different representations, different classification performance across
datasets, and different robustness properties to certain types of image
perturbations. Our findings indicate a remarkable possible synergy across
backbones by leveraging their respective strengths. In principle,
classification accuracy could be improved by over 40 percentage with an
informed selection of the optimal backbone per test example.Using this insight,
we develop a straightforward yet powerful approach to adaptively ensemble
multiple backbones. The approach uses as few as one labeled example per class
to tune the adaptive combination of backbones. On a large collection of
datasets, the method achieves a remarkable increase in accuracy of up to 39.1%
over the best single backbone, well beyond traditional ensembles

对比性语言 - 图像预训练 (CLIP) 是一种重要的图像表示学习方法，本文探讨了不同的 CLIP 训练视觉骨干网络之间的差异，发现它们具有不同的表达方式、在数据集上具有不同的分类性能，以及对某些图像扰动的鲁棒性不同。研究结果表明，在每个测试样例中根据情况选择最佳骨干网络，分类准确性可能提高 40 个百分点以上。基于这一发现，我们提出了一种简单而强大的自适应集成多个骨干网络的方法，该方法可以使用较少数量的已标记样例来调整骨干网络的自适应组合。在大量数据集上，该方法比最佳单一骨干网络的准确性提高了 39.1%，远远超过传统的集成方法。