Decades of research indicate that emotion recognition is more effective when
drawing information from multiple modalities. But what if some modalities are
sometimes missing? To address this problem, we propose a novel
Transformer-based architecture for recognizing valence and arousal in a
time-continuous manner even with missing input modalities. We use a coupling of
cross-attention and self-attention mechanisms to emphasize relationships
between modalities during time and enhance the learning process on weak salient
inputs. Experimental results on the Ulm-TSST dataset show that our model
exhibits an improvement of the concordance correlation coefficient evaluation
of 37% when predicting arousal values and 30% when predicting valence values,
compared to a late-fusion baseline approach.

提出了一种基于 Transformer 的架构，用于识别情绪的正性和负性，即使输入模态缺失，该模型通过交叉关注和自关注机制强调模态之间的关系，提高了对弱显著输入的学习过程，并在 Ul-TSST 数据集上实验，与后期融合基准方法相比，预测唤醒值的一致性相关系数的评估提高了 37％，预测价值的一致性相关系数的评估提高了 30%。