Multimodal emotion recognition (MER), leveraging speech and text, has emerged as a pivotal domain within human-computer interaction, demanding sophisticated methods for effective multimodal integration. The challenge of aligning features across these modalities is significant, with most existing approaches adopting a singular alignment strategy. Such a narrow focus not only limits model performance but also fails to address the complexity and ambiguity inherent in emotional expressions. In response, this paper introduces a Multi-Granularity Cross-Modal Alignment (MGCMA) framework, distinguished by its comprehensive approach encompassing distribution-based, instance-based, and token-based alignment modules. This framework enables a multi-level perception of emotional information across modalities. Our experiments on IEMOCAP demonstrate that our proposed method outperforms current state-of-the-art techniques.

本研究针对现有多模态情感识别中单一对齐策略的限制，提出了一种多粒度跨模态对齐框架（MGCMA），以全面整合情感信息。通过分布式、实例化和基于标记的对齐模块，我们的方法在IEMOCAP数据集上的实验结果显示出显著优于当前最先进的技术，显示了其在情感表达复杂性处理中的潜在影响。

通过多粒度跨模态对齐增强多模态情感识别