In video-based emotion recognition, audio and visual modalities are often
expected to have a complementary relationship, which is widely explored using
cross-attention. However, they may also exhibit weak complementary
relationships, resulting in poor representations of audio-visual features, thus
degrading the performance of the system. To address this issue, we propose
Dynamic Cross-Attention (DCA) that can dynamically select cross-attended or
unattended features on the fly based on their strong or weak complementary
relationship with each other, respectively. Specifically, a simple yet
efficient gating layer is designed to evaluate the contribution of the
cross-attention mechanism and choose cross-attended features only when they
exhibit a strong complementary relationship, otherwise unattended features. We
evaluate the performance of the proposed approach on the challenging RECOLA and
Aff-Wild2 datasets. We also compare the proposed approach with other variants
of cross-attention and show that the proposed model consistently improves the
performance on both datasets.

视频情绪识别中，音频和视觉模态通常被期望具有互补关系，本研究提出了动态交叉注意力（DCA）模型来动态选择跨模态特征的交叉注视或不注视，以优化音频 - 视觉特征的表示，从而提升系统性能。在 RECOLA 和 Aff-Wild2 两个具有挑战性的数据集上，我们评估了所提出方法的性能，并将其与其他交叉注意力的变体进行对比，从而证明了所提出模型在两个数据集上的稳定性能提升。

跨注意力并非必需：音频 - 视觉维度情感识别的动态跨注意力

Cross-Attention is Not Always Needed: Dynamic Cross-Attention for  Audio-Visual Dimensional Emotion Recognition

In video-based emotion recognition (ER), it is important to effectively
leverage the complementary relationship among audio (A) and visual (V)
modalities, while retaining the intra-modal characteristics of individual
modalities. In this paper, a recursive joint attention model is proposed along
with long short-term memory (LSTM) modules for the fusion of vocal and facial
expressions in regression-based ER. Specifically, we investigated the
possibility of exploiting the complementary nature of A and V modalities using
a joint cross-attention model in a recursive fashion with LSTMs to capture the
intra-modal temporal dependencies within the same modalities as well as among
the A-V feature representations. By integrating LSTMs with recursive joint
cross-attention, our model can efficiently leverage both intra- and inter-modal
relationships for the fusion of A and V modalities. The results of extensive
experiments performed on the challenging Affwild2 and Fatigue (private)
datasets indicate that the proposed A-V fusion model can significantly
outperform state-of-art-methods.

本文提出了一种递归联合注意模型，结合长短期记忆模块，用于融合语音和面部表情进行基于回归的情感识别，结果表明该模型比现有技术表现更好。