In video-based emotion recognition (ER), it is important to effectively
leverage the complementary relationship among audio (A) and visual (V)
modalities, while retaining the intra-modal characteristics of individual
modalities. In this paper, a recursive joint attention model is proposed along
with long short-term memory (LSTM) modules for the fusion of vocal and facial
expressions in regression-based ER. Specifically, we investigated the
possibility of exploiting the complementary nature of A and V modalities using
a joint cross-attention model in a recursive fashion with LSTMs to capture the
intra-modal temporal dependencies within the same modalities as well as among
the A-V feature representations. By integrating LSTMs with recursive joint
cross-attention, our model can efficiently leverage both intra- and inter-modal
relationships for the fusion of A and V modalities. The results of extensive
experiments performed on the challenging Affwild2 and Fatigue (private)
datasets indicate that the proposed A-V fusion model can significantly
outperform state-of-art-methods.

本文提出了一种递归联合注意模型，结合长短期记忆模块，用于融合语音和面部表情进行基于回归的情感识别，结果表明该模型比现有技术表现更好。