The study of human emotions, traditionally a cornerstone in fields like
psychology and neuroscience, has been profoundly impacted by the advent of
artificial intelligence (AI). Multiple channels, such as speech (voice) and
facial expressions (image), are crucial in understanding human emotions.
However, AI's journey in multimodal emotion recognition (MER) is marked by
substantial technical challenges. One significant hurdle is how AI models
manage the absence of a particular modality - a frequent occurrence in
real-world situations. This study's central focus is assessing the performance
and resilience of two strategies when confronted with the lack of one modality:
a novel multimodal dynamic modality and view selection and a cross-attention
mechanism. Results on the RECOLA dataset show that dynamic selection-based
methods are a promising approach for MER. In the missing modalities scenarios,
all dynamic selection-based methods outperformed the baseline. The study
concludes by emphasizing the intricate interplay between audio and video
modalities in emotion prediction, showcasing the adaptability of dynamic
selection methods in handling missing modalities.

这项研究的重点是评估两种策略在缺少一个感知模态的情况下的表现和韧性：一种新颖的多模式动态模态和视角选择方法以及一种交叉注意机制。RECOLA 数据集上的结果表明动态选择方法是一种有前景的多模态情感识别方法。在缺失模态的场景下，所有基于动态选择的方法都优于基准测试。研究总结强调了音频和视频模态在情感预测中的复杂相互作用，展示了动态选择方法在处理缺失模态方面的适应性。