Most current audio-visual emotion recognition models lack the flexibility
needed for deployment in practical applications. We envision a multimodal
system that works even when only one modality is available and can be
implemented interchangeably for either predicting emotional attributes or
recognizing categorical emotions. Achieving such flexibility in a multimodal
emotion recognition system is difficult due to the inherent challenges in
accurately interpreting and integrating varied data sources. It is also a
challenge to robustly handle missing or partial information while allowing
direct switch between regression and classification tasks. This study proposes
a \emph{versatile audio-visual learning} (VAVL) framework for handling unimodal
and multimodal systems for emotion regression and emotion classification tasks.
We implement an audio-visual framework that can be trained even when audio and
visual paired data is not available for part of the training set (i.e., audio
only or only video is present). We achieve this effective representation
learning with audio-visual shared layers, residual connections over shared
layers, and a unimodal reconstruction task. Our experimental results reveal
that our architecture significantly outperforms strong baselines on both the
CREMA-D and MSP-IMPROV corpora. Notably, VAVL attains a new state-of-the-art
performance in the emotional attribute prediction task on the MSP-IMPROV
corpus. Code available at: this https URL

提出了用于处理单模式和多模式情感情况的多功能音视频学习框架，通过音视频共享层，残差连接和单模态重构任务实现有效的表示学习，在情感属性预测任务上达到了新的最先进性能。