Multimodal fusion breaks through the barriers between diverse modalities and
has already yielded numerous impressive performances. However, in various
specialized fields, it is struggling to obtain sufficient alignment data for
the training process, which seriously limits the use of previously elegant
models. Thus, semi-supervised learning attempts to achieve multimodal alignment
with fewer matched pairs but traditional methods like pseudo-labeling are
difficult to apply in domains with no label information. To address these
problems, we transform semi-supervised multimodal alignment into a manifold
matching problem and propose a new method based on CLIP, named Gentle-CLIP.
Specifically, we design a novel semantic density distribution loss to explore
implicit semantic alignment information from unpaired multimodal data by
constraining the latent representation distribution with fine granularity, thus
eliminating the need for numerous strictly matched pairs. Meanwhile, we
introduce multi-kernel maximum mean discrepancy as well as self-supervised
contrastive loss to pull separate modality distributions closer and enhance the
stability of the representation distribution. In addition, the contrastive loss
used in CLIP is employed on the supervised matched data to prevent negative
optimization. Extensive experiments conducted on a range of tasks in various
fields, including protein, remote sensing, and the general vision-language
field, demonstrate the effectiveness of our proposed Gentle-CLIP.

使用 Gentle-CLIP 方法，通过将半监督多模态对齐转化为流形匹配问题，利用新型的语义密度分布损失、多核最大平均差异和自监督对比损失等技术实现了更少匹配对的多模态对齐，提升了表示分布的稳定性和模态间的距离，且在蛋白质、遥感和图像语言领域的多个任务上得到了验证。