This paper considers contrastive training for cross-modal 0-shot transfer
wherein a pre-trained model in one modality is used for representation learning
in another domain using pairwise data. The learnt models in the latter domain
can then be used for a diverse set of tasks in a zero-shot way, similar to
``Contrastive Language-Image Pre-training (CLIP)'' and ``Locked-image Tuning
(LiT)'' that have recently gained considerable attention. Most existing works
for cross-modal representation alignment (including CLIP and LiT) use the
standard contrastive training objective, which employs sets of positive and
negative examples to align similar and repel dissimilar training data samples.
However, similarity amongst training examples has a more continuous nature,
thus calling for a more `non-binary' treatment. To address this, we propose a
novel loss function called Continuously Weighted Contrastive Loss (CWCL) that
employs a continuous measure of similarity. With CWCL, we seek to align the
embedding space of one modality with another. Owing to the continuous nature of
similarity in the proposed loss function, these models outperform existing
methods for 0-shot transfer across multiple models, datasets and modalities.
Particularly, we consider the modality pairs of image-text and speech-text and
our models achieve 5-8% (absolute) improvement over previous state-of-the-art
methods in 0-shot image classification and 20-30% (absolute) improvement in
0-shot speech-to-intent classification and keyword classification.

本文探讨了交叉感知培训中的对比训练，其中一个模态的预训练模型通过成对的数据用于学习另一个领域的表示学习。采用连续加权对比损失（CWCL）的本文提出了一个新的损失函数，通过连续相似性度量来对齐嵌入空间。在 0-shot 图像分类、0-shot 语音意图分类和关键词分类中取得了相对于现有方法的显著改进。