Previous work on multimodal sentence embedding has proposed multimodal
contrastive learning and achieved promising results. However, by taking the
rest of the batch as negative samples without reviewing when forming
contrastive pairs, those studies encountered many suspicious and noisy negative
examples, significantly affecting the methods' overall performance. In this
work, we propose KDMCSE (Knowledge Distillation Multimodal contrastive learning
of Sentence Embeddings), a novel approach that enhances the discrimination and
generalizability of multimodal representation and inherits the knowledge from
the teacher model to learn the difference between positive and negative
instances and via that, can detect noisy and wrong negative samples effectively
before they are calculated in the contrastive objective. Furthermore, to
overcome the limitation of modeling the variation within negative pairs, we
introduce a new contrastive objective, AdapACSE (Adaptive Angular Margin
Supervised Contrastive Learning for Multimodal sentence embeddings), that
enhances the discriminative representation by strengthening the margin within
the angular space while capturing varying semantics within the negative.
Experimental results on widely used Semantic Textual Similarity (STS)
benchmarks demonstrate the effectiveness of our approach.

本研究提出了 KDMCSE，一种基于知识蒸馏的多模态对比学习方法，通过利用教师模型的知识来增强多模态表示的区分性和泛化能力，有效地检测和消除有噪声和错误的负样本，同时引入了 AdapACSE，一种新的自适应角度间隔有监督对比学习方法，增强了在角度空间中的边界，捕捉了负样本中不同的语义变化。在广泛使用的语义文本相似性基准测试中，实验证明了我们方法的有效性。