Multimodal emotion recognition (MER) in practical scenarios presents a
significant challenge due to the presence of incomplete data, such as missing
or noisy data. Traditional methods often discard missing data or replace it
with a zero vector, neglecting the availability issue of noisy data.
Consequently, these approaches are not fully applicable to realistic scenarios,
where both missing and noisy data are prevalent. To address this problem, we
propose a novel noise-robust MER model, named NMER, which effectively learns
robust multimodal joint representations from incomplete data containing noise.
Our approach incorporates two key components. First, we introduce a noise
scheduler that adjusts the type and level of noise in the training data,
emulating the characteristics of incomplete data in realistic scenarios.
Second, we employ a Variational AutoEncoder (VAE)-based NMER model to generate
robust multimodal joint representations from the noisy data, leveraging the
modality invariant feature. The experimental results on the benchmark dataset
IEMOCAP indicate the proposed NMER outperforms state-of-the-art MER systems.
The ablation results also confirm the effectiveness of the VAE structure. We
release our code at \href{this https URL

通过引入噪声和可变自动编码器（Variational AutoEncoder）结构，提出了一种新的噪声鲁棒多模情感识别模型（NMER model），能够有效地从包含噪声的不完整数据中学习到鲁棒的多模态联合表示，实验证明 NMER 在 IEMOCAP 数据集上胜过最先进的多模情感识别系统。