Recently, multimodal contrastive learning (MMCL) approaches, such as CLIP,
have achieved a remarkable success in learning representations that are robust
against distribution shift and generalize to new domains. Despite the empirical
success, the mechanism behind learning such generalizable representations is
not understood. In this work, we rigorously analyze this problem and uncover
two mechanisms behind MMCL's robustness: \emph{intra-class contrasting}, which
allows the model to learn features with a high variance, and \emph{inter-class
feature sharing}, where annotated details in one class help learning other
classes better. Both mechanisms prevent spurious features that are
over-represented in the training data to overshadow the generalizable core
features. This yields superior zero-shot classification accuracy under
distribution shift. Furthermore, we theoretically demonstrate the benefits of
using rich captions on robustness and explore the effect of annotating
different types of details in the captions. We validate our theoretical
findings through experiments, including a well-designed synthetic experiment
and an experiment involving training CLIP on MS COCO and evaluating the model
on variations of shifted ImageNet.

通过分析多模态对比学习 (MMCL) 方法，我们揭示了其在学习鲁棒表示方面的两种机制：类内对比机制和跨类特征共享机制，这两种机制防止了训练数据中过度呈现的虚假特征对通用核心特征的影响，从而在分布偏移下实现了更优的零样本分类精度。