Multimodal fusion breaks through the barriers between diverse modalities and
has already yielded numerous impressive performances. However, in various
specialized fields, it is struggling to obtain sufficient alignment data for
the training process, which seriously limits the use of previously elegant
models. Thus, semi-supervised learning attempts to achieve multimodal alignment
with fewer matched pairs but traditional methods like pseudo-labeling are
difficult to apply in domains with no label information. To address these
problems, we transform semi-supervised multimodal alignment into a manifold
matching problem and propose a new method based on CLIP, named Gentle-CLIP.
Specifically, we design a novel semantic density distribution loss to explore
implicit semantic alignment information from unpaired multimodal data by
constraining the latent representation distribution with fine granularity, thus
eliminating the need for numerous strictly matched pairs. Meanwhile, we
introduce multi-kernel maximum mean discrepancy as well as self-supervised
contrastive loss to pull separate modality distributions closer and enhance the
stability of the representation distribution. In addition, the contrastive loss
used in CLIP is employed on the supervised matched data to prevent negative
optimization. Extensive experiments conducted on a range of tasks in various
fields, including protein, remote sensing, and the general vision-language
field, demonstrate the effectiveness of our proposed Gentle-CLIP.

使用 Gentle-CLIP 方法，通过将半监督多模态对齐转化为流形匹配问题，利用新型的语义密度分布损失、多核最大平均差异和自监督对比损失等技术实现了更少匹配对的多模态对齐，提升了表示分布的稳定性和模态间的距离，且在蛋白质、遥感和图像语言领域的多个任务上得到了验证。

软对齐下探索低质量多模态数据中的对齐语义

Gentle-CLIP: Exploring Aligned Semantic In Low-Quality Multimodal Data  With Soft Alignment

Fusion and inference from multiple and massive disparate data sources - the
requirement for our most challenging data analysis problems and the goal of our
most ambitious statistical pattern recognition methodologies - -has many and
varied aspects which are currently the target of intense research and
development. One aspect of the overall challenge is manifold matching -
identifying embeddings of multiple disparate data spaces into the same
low-dimensional space where joint inference can be pursued. We investigate this
manifold matching task from the perspective of jointly optimizing the fidelity
of the embeddings and their commensurability with one another, with a specific
statistical inference exploitation task in mind. Our results demonstrate when
and why our joint optimization methodology is superior to either version of
separate optimization. The methodology is illustrated with simulations and an
application in document matching.

研究多源异构数据的匹配问题，以实现嵌入式统计推断任务的联合优化，并用文档匹配的应用程序演示了该方法的有效性。