This paper presents a novel approach to processing multimodal data for
dynamic emotion recognition, named as the Multimodal Masked Autoencoder for
Dynamic Emotion Recognition (MultiMAE-DER). The MultiMAE-DER leverages the
closely correlated representation information within spatiotemporal sequences
across visual and audio modalities. By utilizing a pre-trained masked
autoencoder model, the MultiMAEDER is accomplished through simple,
straightforward finetuning. The performance of the MultiMAE-DER is enhanced by
optimizing six fusion strategies for multimodal input sequences. These
strategies address dynamic feature correlations within cross-domain data across
spatial, temporal, and spatiotemporal sequences. In comparison to
state-of-the-art multimodal supervised learning models for dynamic emotion
recognition, MultiMAE-DER enhances the weighted average recall (WAR) by 4.41%
on the RAVDESS dataset and by 2.06% on the CREMAD. Furthermore, when compared
with the state-of-the-art model of multimodal self-supervised learning,
MultiMAE-DER achieves a 1.86% higher WAR on the IEMOCAP dataset.

本文提出了一种名为多模态蒙特卡洛自动编码器（MultiMAE-DER）的处理多模态数据用于动态情感识别的新方法。MultiMAE-DER 利用视觉和音频模态之间的时空序列中紧密相关的表示信息。通过利用预训练的蒙特卡洛自动编码器模型，通过简单直接的微调实现 MultiMAE-DER。该方法通过优化六种多模态输入序列融合策略来提高 MultiMAE-DER 的性能，解决了跨域数据中的动态特征相关性问题。与现有的多模态监督学习模型相比，MultiMAE-DER 在 RAVDESS 数据集上的加权平均召回率（WAR）提高了 4.41％，在 CREMAD 上提高了 2.06％。此外，与多模态自监督学习的最新模型相比，MultiMAE-DER 在 IEMOCAP 数据集上的 WAR 提高了 1.86％。