Image-text contrastive models such as CLIP are useful for a variety of
downstream applications including zero-shot classification, image-text
retrieval and transfer learning. However, these contrastively trained
vision-language models often fail on compositional visio-linguistic tasks such
as Winoground with performance equivalent to random chance. In our paper, we
address this issue and propose a sample-efficient light-weight method called
SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
of CLIP. The core idea of our method is to use differentiable image
parameterizations to fine-tune CLIP with a distillation objective from large
text-to-image generative models such as Stable-Diffusion which are relatively
good at visio-linguistic reasoning tasks. On the challenging Winoground
compositional reasoning benchmark, our method improves the absolute
visio-linguistic performance of different CLIP models by up to 7%, while on the
ARO dataset, our method improves the visio-linguistic performance by upto 3%.
As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
that the zero-shot performance improves marginally on a variety of downstream
datasets. Our method reinforces that carefully designed distillation objectives
from generative models can be leveraged to extend existing contrastive
image-text models with improved visio-linguistic reasoning capabilities.

通过使用不同的图像参数化方法，利用生成模型和巧妙设计的蒸馏目标，我们提出了一种轻量级而高效的方法 SDS-CLIP，来改善 CLIP 模型的组合视觉 - 语言推理能力并在多个数据集上显示了显著的性能提升。