Audiovisual emotion recognition (ER) in videos has immense potential over
unimodal performance. It effectively leverages the inter- and intra-modal
dependencies between visual and auditory modalities. This work proposes a novel
audio-visual emotion recognition system utilizing a joint multimodal
transformer architecture with key-based cross-attention. This framework aims to
exploit the complementary nature of audio and visual cues (facial expressions
and vocal patterns) in videos, leading to superior performance compared to
solely relying on a single modality. The proposed model leverages separate
backbones for capturing intra-modal temporal dependencies within each modality
(audio and visual). Subsequently, a joint multimodal transformer architecture
integrates the individual modality embeddings, enabling the model to
effectively capture inter-modal (between audio and visual) and intra-modal
(within each modality) relationships. Extensive evaluations on the challenging
Affwild2 dataset demonstrate that the proposed model significantly outperforms
baseline and state-of-the-art methods in ER tasks.

通过使用多模态 Transformer 架构，全面利用视频中视觉和听觉模态之间的交互和内部依赖关系，以识别音频视觉表情和声音模式，该模型在 Affwild2 数据集上表现出优越性能。

野外情感维度识别的联合多模态变压器

Joint Multimodal Transformer for Dimensional Emotional Recognition in  the Wild

This paper presents a light-weight and accurate deep neural model for
audiovisual emotion recognition. To design this model, the authors followed a
philosophy of simplicity, drastically limiting the number of parameters to
learn from the target datasets, always choosing the simplest earning methods:
i) transfer learning and low-dimensional space embedding allows to reduce the
dimensionality of the representations. ii) The isual temporal information is
handled by a simple score-per-frame selection process, averaged across time.
iii) A simple frame selection echanism is also proposed to weight the images of
a sequence. iv) The fusion of the different modalities is performed at
prediction level (late usion). We also highlight the inherent challenges of the
AFEW dataset and the difficulty of model selection with as few as 383
validation equences. The proposed real-time emotion classifier achieved a
state-of-the-art accuracy of 60.64 % on the test set of AFEW, and ranked 4th at
he Emotion in the Wild 2018 challenge.

该研究提出了一种轻量级和准确的深度神经模型，通过迁移学习、降维和简单的帧选择机制实现音视频情感识别，取得了情感识别领域 State-of-the-art 的准确率。