Most current audio-visual emotion recognition models lack the flexibility
needed for deployment in practical applications. We envision a multimodal
system that works even when only one modality is available and can be
implemented interchangeably for either predicting emotional attributes or
recognizing categorical emotions. Achieving such flexibility in a multimodal
emotion recognition system is difficult due to the inherent challenges in
accurately interpreting and integrating varied data sources. It is also a
challenge to robustly handle missing or partial information while allowing
direct switch between regression and classification tasks. This study proposes
a \emph{versatile audio-visual learning} (VAVL) framework for handling unimodal
and multimodal systems for emotion regression and emotion classification tasks.
We implement an audio-visual framework that can be trained even when audio and
visual paired data is not available for part of the training set (i.e., audio
only or only video is present). We achieve this effective representation
learning with audio-visual shared layers, residual connections over shared
layers, and a unimodal reconstruction task. Our experimental results reveal
that our architecture significantly outperforms strong baselines on both the
CREMA-D and MSP-IMPROV corpora. Notably, VAVL attains a new state-of-the-art
performance in the emotional attribute prediction task on the MSP-IMPROV
corpus. Code available at: this https URL

提出了用于处理单模式和多模式情感情况的多功能音视频学习框架，通过音视频共享层，残差连接和单模态重构任务实现有效的表示学习，在情感属性预测任务上达到了新的最先进性能。

情感回归和分类任务中处理单一和多模态的多功能视听学习

Versatile Audio-Visual Learning for Handling Single and Multi Modalities  in Emotion Regression and Classification Tasks

We propose VADEC, a multi-task framework that exploits the correlation
between the categorical and dimensional models of emotion representation for
better subjectivity analysis. Focusing primarily on the effective detection of
emotions from tweets, we jointly train multi-label emotion classification and
multi-dimensional emotion regression, thereby utilizing the inter-relatedness
between the tasks. Co-training especially helps in improving the performance of
the classification task as we outperform the strongest baselines with 3.4%,
11%, and 3.9% gains in Jaccard Accuracy, Macro-F1, and Micro-F1 scores
respectively on the AIT dataset. We also achieve state-of-the-art results with
11.3% gains averaged over six different metrics on the SenWave dataset. For the
regression task, VADEC, when trained with SenWave, achieves 7.6% and 16.5%
gains in Pearson Correlation scores over the current state-of-the-art on the
EMOBANK dataset for the Valence (V) and Dominance (D) affect dimensions
respectively. We conclude our work with a case study on COVID-19 tweets posted
by Indians that further helps in establishing the efficacy of our proposed
solution.

提出了一种多任务框架 VADEC，用于更好地进行情感分析，其中包括多标签情感分类和多维情感回归，并通过协同训练利用任务之间的相互关系，其性能超过了最强基线，尤其是在情感分类任务方面，为 AIT 数据集中的 Jaccard Accuracy、Macro-F1 和 Micro-F1 得分分别带来了 3.4%、11% 和 3.9% 的提高，在 SenWave 数据集上表现出了 11.3% 的平均增益。当在 SenWave 数据集上训练时，VADEC 对于 Valence（V）和 Dominance（D）情感维度在 EMOBANK 数据集上实现了 7.6% 和 16.5% 的皮尔逊相关系数增益，最后在印度发布的 COVID-19 tweet 的案例研究中证明了该框架的有效性。