Audiovisual emotion recognition (ER) in videos has immense potential over
unimodal performance. It effectively leverages the inter- and intra-modal
dependencies between visual and auditory modalities. This work proposes a novel
audio-visual emotion recognition system utilizing a joint multimodal
transformer architecture with key-based cross-attention. This framework aims to
exploit the complementary nature of audio and visual cues (facial expressions
and vocal patterns) in videos, leading to superior performance compared to
solely relying on a single modality. The proposed model leverages separate
backbones for capturing intra-modal temporal dependencies within each modality
(audio and visual). Subsequently, a joint multimodal transformer architecture
integrates the individual modality embeddings, enabling the model to
effectively capture inter-modal (between audio and visual) and intra-modal
(within each modality) relationships. Extensive evaluations on the challenging
Affwild2 dataset demonstrate that the proposed model significantly outperforms
baseline and state-of-the-art methods in ER tasks.

通过使用多模态 Transformer 架构，全面利用视频中视觉和听觉模态之间的交互和内部依赖关系，以识别音频视觉表情和声音模式，该模型在 Affwild2 数据集上表现出优越性能。