Multi-modal semantic understanding requires integrating information from
different modalities to extract users' real intention behind words. Most
previous work applies a dual-encoder structure to separately encode image and
text, but fails to learn cross-modal feature alignment, making it hard to
achieve cross-modal deep information interaction. This paper proposes a novel
CLIP-guided contrastive-learning-based architecture to perform multi-modal
feature alignment, which projects the features derived from different
modalities into a unified deep space. On multi-modal sarcasm detection (MMSD)
and multi-modal sentiment analysis (MMSA) tasks, the experimental results show
that our proposed model significantly outperforms several baselines, and our
feature alignment strategy brings obvious performance gain over models with
different aggregating methods and models even enriched with knowledge. More
importantly, our model is simple to implement without using task-specific
external knowledge, and thus can easily migrate to other multi-modal tasks. Our
source codes are available at this https URL

本篇研究提出了一种基于 CLIP 引导的对比学习的架构，用于执行多模态特征对齐，将来自不同模态的特征投影到一个统一的深度空间，实验结果表明，我们提出的模型在多模态讽刺检测和多模态情感分析任务中明显优于多个基准模型，我们的特征对齐策略相对于其他聚合方法和甚至富含知识的模型也带来了明显的性能增益，此外，我们的模型实现简单，无需使用特定任务的外部知识，因此可以轻松迁移到其他多模态任务。