CLIP has become a promising language-supervised visual pre-training framework
and achieves excellent performance over a wide range of tasks. This paper aims
to distill small CLIP models supervised by a large teacher CLIP model. We
propose several distillation strategies, including relation, feature, gradient
and contrastive paradigm, to examine the impact on CLIP distillation. We show
that the simplest feature mimicry with MSE loss performs best. Moreover,
interactive contrastive learning and relation-based distillation are also
critical in performance improvement. We apply the unified method to distill
several student networks trained on 15 million (image, text) pairs.
Distillation improves the student CLIP models consistently over zero-shot
ImageNet classification and cross-modal retrieval benchmarks. We hope our
empirical study will become an important baseline for future CLIP distillation
research. The code is available at https://github.com/winycg/CLIP-KD.

通过使用关系、特征、梯度和对比范式等多种蒸馏策略对基于大型教师 CLIP 模型的小型 CLIP 模型进行监督，本研究在零样本 ImageNet 分类和跨模态检索基准上一致提升了学生 CLIP 模型的性能，提供了重要的 CLIP 蒸馏研究基准。