Multi-modal learning has become increasingly popular due to its ability to
leverage information from different data sources (e.g., text and images) to
improve the model performance. Recently, CLIP has emerged as an effective
approach that employs vision-language contrastive pretraining to learn joint
image and text representations and exhibits remarkable performance in zero-shot
learning and text-guided natural image generation. Despite the huge practical
success of CLIP, its theoretical understanding remains elusive. In this paper,
we formally study transferrable representation learning underlying CLIP and
demonstrate how features from different modalities get aligned. We also analyze
its zero-shot transfer performance on the downstream tasks. Inspired by our
analysis, we propose a new CLIP-type approach, which achieves better
performance than CLIP and other state-of-the-art methods on benchmark datasets.

通过对 CLIP 的理论研究，我们证明了多模态学习的可转移表示学习，并分析了其在零样本学习和下游任务中的性能。在此基础上，我们提出了一种新的 CLIP 类型方法，在基准数据集上实现了比 CLIP 和其他最先进方法更好的性能。