Contrastive Language-Image Pretraining (CLIP) performs zero-shot image classification by mapping images and textual class representation into a shared embedding space, then retrieving the class closest to the image. This work provides a new approach for interpreting CLIP models for image classification from the lens of mutual knowledge between the two modalities. Specifically, we ask: what concepts do both vision and language CLIP encoders learn in common that influence the joint embedding space, causing points to be closer or further apart? We answer this question via an approach of textual concept-based explanations, showing their effectiveness, and perform an analysis encompassing a pool of 13 CLIP models varying in architecture, size and pretraining datasets. We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions. Our approach demonstrates an effective and human-friendly way of understanding zero-shot classification decisions with CLIP.

本研究解决了CLIP模型在图像分类中的解读与分析问题，尤其是如何理解视觉和语言两个模态之间的共同概念。通过文本概念的解释方法，我们分析了13个不同架构、规模和预训练数据集的CLIP模型，发现它们的互知识关系有效影响了零样本预测的结果。这一方法为理解CLIP的零样本分类决策提供了有效且易于人类理解的方式。

通过互知识解释和分析CLIP的零样本图像分类