With the continuous development of deep learning (DL), the task of multimodal
dialogue emotion recognition (MDER) has recently received extensive research
attention, which is also an essential branch of DL. The MDER aims to identify
the emotional information contained in different modalities, e.g., text, video,
and audio, in different dialogue scenes. However, existing research has focused
on modeling contextual semantic information and dialogue relations between
speakers while ignoring the impact of event relations on emotion. To tackle the
above issues, we propose a novel Dialogue and Event Relation-Aware Graph
Convolutional Neural Network for Multimodal Emotion Recognition (DER-GCN)
method. It models dialogue relations between speakers and captures latent event
relations information. Specifically, we construct a weighted multi-relationship
graph to simultaneously capture the dependencies between speakers and event
relations in a dialogue. Moreover, we also introduce a Self-Supervised Masked
Graph Autoencoder (SMGAE) to improve the fusion representation ability of
features and structures. Next, we design a new Multiple Information Transformer
(MIT) to capture the correlation between different relations, which can provide
a better fuse of the multivariate information between relations. Finally, we
propose a loss optimization strategy based on contrastive learning to enhance
the representation learning ability of minority class features. We conduct
extensive experiments on the IEMOCAP and MELD benchmark datasets, which verify
the effectiveness of the DER-GCN model. The results demonstrate that our model
significantly improves both the average accuracy and the f1 value of emotion
recognition.

提出了一种新颖的基于对话和事件关系感知的图卷积神经网络用于多模态情感识别，该方法通过建立加权多关系图来捕捉说话者之间的对话关系和事件关系，设计了多重信息变换器以捕捉不同关系之间的相关性，通过对比学习的损失优化策略增强了少数类特征的表示学习能力。实验证明，DER-GCN 模型在情感识别方面显著提高了平均准确率和 F1 值。