Multi-modal emotion recognition has recently gained a lot of attention since
it can leverage diverse and complementary relationships over multiple
modalities, such as audio, visual, and text. Most state-of-the-art methods for
multimodal fusion rely on recurrent networks or conventional attention
mechanisms that do not effectively leverage the complementary nature of the
modalities. In this paper, we focus on dimensional emotion recognition based on
the fusion of facial, vocal, and text modalities extracted from videos.
Specifically, we propose a recursive cross-modal attention (RCMA) to
effectively capture the complementary relationships across the modalities in a
recursive fashion. The proposed model is able to effectively capture the
inter-modal relationships by computing the cross-attention weights across the
individual modalities and the joint representation of the other two modalities.
To further improve the inter-modal relationships, the obtained attended
features of the individual modalities are again fed as input to the cross-modal
attention to refine the feature representations of the individual modalities.
In addition to that, we have used Temporal convolution networks (TCNs) to
capture the temporal modeling (intra-modal relationships) of the individual
modalities. By deploying the TCNs as well cross-modal attention in a recursive
fashion, we are able to effectively capture both intra- and inter-modal
relationships across the audio, visual, and text modalities. Experimental
results on validation-set videos from the AffWild2 dataset indicate that our
proposed fusion model is able to achieve significant improvement over the
baseline for the sixth challenge of Affective Behavior Analysis in-the-Wild
2024 (ABAW6) competition.

通过跨模态注意力和时间卷积网络，提出了一种用于多模态情感识别的模型，能够有效地捕捉不同模态之间的互补关系，并在赛事中实现显著改进。

情感维度识别中的递归跨模态注意力的多模态融合

Recursive Cross-Modal Attention for Multimodal Fusion in Dimensional  Emotion Recognition

We consider the task of dimensional emotion recognition on video data using
deep learning. While several previous methods have shown the benefits of
training temporal neural network models such as recurrent neural networks
(RNNs) on hand-crafted features, few works have considered combining
convolutional neural networks (CNNs) with RNNs. In this work, we present a
system that performs emotion recognition on video data using both CNNs and
RNNs, and we also analyze how much each neural network component contributes to
the system's overall performance. We present our findings on videos from the
Audio/Visual+Emotion Challenge (AV+EC2015). In our experiments, we analyze the
effects of several hyperparameters on overall performance while also achieving
superior performance to the baseline and other competing methods.

本文通过使用卷积神经网络和循环神经网络来识别视频数据中的情感，分析两种神经网络对系统性能的贡献，并在实验中实现了优于基线和其他竞争方法的表现。