Emotion Recognition in Conversation (ERC) plays an important role in driving
the development of human-machine interaction. Emotions can exist in multiple
modalities, and multimodal ERC mainly faces two problems: (1) the noise problem
in the cross-modal information fusion process, and (2) the prediction problem
of less sample emotion labels that are semantically similar but different
categories. To address these issues and fully utilize the features of each
modality, we adopted the following strategies: first, deep emotion cues
extraction was performed on modalities with strong representation ability, and
feature filters were designed as multimodal prompt information for modalities
with weak representation ability. Then, we designed a Multimodal Prompt
Transformer (MPT) to perform cross-modal information fusion. MPT embeds
multimodal fusion information into each attention layer of the Transformer,
allowing prompt information to participate in encoding textual features and
being fused with multi-level textual information to obtain better multimodal
fusion features. Finally, we used the Hybrid Contrastive Learning (HCL)
strategy to optimize the model's ability to handle labels with few samples.
This strategy uses unsupervised contrastive learning to improve the
representation ability of multimodal fusion and supervised contrastive learning
to mine the information of labels with few samples. Experimental results show
that our proposed model outperforms state-of-the-art models in ERC on two
benchmark datasets.

多模 ERC 在驱动人机交互发展方面扮演了重要角色，但面临多模态信息融合中的噪声问题以及类别语义相似但样本标签较少的预测问题。为应对这些问题，本研究采用了深度情感线索提取、特征过滤器设计、多模态提示转换器和混合对比学习等策略，实现了更好的多模态融合表现和对少样本标签的处理能力。实验结果证明，所提出的模型在两个基准数据集上优于现有模型。