Vision-language models, such as CLIP, have shown promising
Out-of-Distribution (OoD) generalization under various types of distribution
shifts. Recent studies attempted to investigate the leading cause of this
capability. In this work, we follow the same path, but focus on a specific type
of OoD data - images with novel compositions of attribute-object pairs - and
study whether such models can successfully classify those images into
composition classes. We carefully designed an authentic image test dataset
called ImageNet-AO, consisting of attributes for objects that are unlikely
encountered in the CLIP training sets. We found that CLIPs trained with large
datasets such as OpenAI CLIP, LAION-400M, and LAION-2B show orders-of-magnitude
improvement in effective compositional OoD generalization compared to both
supervised models and CLIPs trained with smaller datasets, such as CC-12M and
YFCC-15M. Our results provide evidence that the scale and diversity of training
data and language supervision play a key role in unlocking the compositional
generalization abilities of vision-language models.

通过研究图像与语言模型 (CLIP) 在不同类型的分布转换下的通用性，重点关注图像与语言模型在新颖的属性 - 对象配对组合中的分类能力。研究表明，使用大规模的训练数据和语言监督可以显著提高视觉 - 语言模型的组合泛化能力。

语言在 CLIP 对象 - 属性组合泛化中起着关键作用

Language Plays a Pivotal Role in the Object-Attribute Compositional  Generalization of CLIP

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen
attribute-object pairs based on a limited set of observed examples. Current
CZSL methodologies, despite their advancements, tend to neglect the distinct
specificity levels present in attributes. For instance, given images of sliced
strawberries, they may fail to prioritize `Sliced-Strawberry' over a generic
`Red-Strawberry', despite the former being more informative. They also suffer
from ballooning search space when shifting from Close-World (CW) to Open-World
(OW) CZSL. To address the issues, we introduce the Context-based and
Diversity-driven Specificity learning framework for CZSL (CDS-CZSL). Our
framework evaluates the specificity of attributes by considering the diversity
of objects they apply to and their related context. This novel approach allows
for more accurate predictions by emphasizing specific attribute-object pairs
and improves composition filtering in OW-CZSL. We conduct experiments in both
CW and OW scenarios, and our model achieves state-of-the-art results across
three datasets.

为了更准确地预测未见过的属性 - 物体对并改善在开放世界中的组合过滤，在通常被忽略的特异性水平上考虑属性的多样性和上下文，我们引入了基于上下文和多样性的特异性学习框架用于组合零样本学习 (CDS-CZSL)。在闭世界和开放世界的情景中进行了实验，我们的模型在三个数据集上实现了最先进的结果。