The research on human emotion under multimedia stimulation based on
physiological signals is an emerging field, and important progress has been
achieved for emotion recognition based on multi-modal signals. However, it is
challenging to make full use of the complementarity among
spatial-spectral-temporal domain features for emotion recognition, as well as
model the heterogeneity and correlation among multi-modal signals. In this
paper, we propose a novel two-stream heterogeneous graph recurrent neural
network, named HetEmotionNet, fusing multi-modal physiological signals for
emotion recognition. Specifically, HetEmotionNet consists of the
spatial-temporal stream and the spatial-spectral stream, which can fuse
spatial-spectral-temporal domain features in a unified framework. Each stream
is composed of the graph transformer network for modeling the heterogeneity,
the graph convolutional network for modeling the correlation, and the gated
recurrent unit for capturing the temporal domain or spectral domain dependency.
Extensive experiments on two real-world datasets demonstrate that our proposed
model achieves better performance than state-of-the-art baselines.

本研究提出了一种新颖的两流异构图递归神经网络，名为 HetEmotionNet，该网络可以融合多模态生理信号用于情感识别，并在两个实际数据集上取得了比现有基准更好的性能。