Contrastive Language--Image Pre-training (CLIP) has manifested remarkable
improvements in zero-shot classification and cross-modal vision-language tasks.
Yet, from a geometrical point of view, the CLIP embedding space has been found
to have a pronounced modality gap. This gap renders the embedding space overly
sparse and disconnected, with different modalities being densely distributed in
distinct subregions of the hypersphere. In this work, we aim at answering two
main questions: 1. Does sharing the parameter space between the multi-modal
encoders reduce the modality gap? 2. Can the gap be mitigated by pushing apart
the uni-modal embeddings via intra-modality separation? We design AlignCLIP, in
order to answer these questions and show that answers to both questions are
positive. Through extensive experiments, we show that AlignCLIP achieves
noticeable enhancements in the cross-modal alignment of the embeddings, and
thereby, reduces the modality gap, while maintaining the performance across
several downstream evaluations, such as zero-shot image classification,
zero-shot multi-modal retrieval and zero-shot semantic text similarity.

通过设计 AlignCLIP，本文回答了共享多模态编码器参数空间是否减少模态间隙以及通过内部模态分离是否可以缓解间隙的两个主要问题，并通过大量实验证明了 AlignCLIP 在嵌入的跨模态对齐上取得了显著的改进，从而减少了模态间隙，并在零样本图像分类、零样本多模态检索和零样本语义文本相似性等多个下游评估中保持了性能。