Leveraging the synergy of both audio data and visual data is essential for
understanding human emotions and behaviors, especially in in-the-wild setting.
Traditional methods for integrating such multimodal information often stumble,
leading to less-than-ideal outcomes in the task of facial action unit
detection. To overcome these shortcomings, we propose a novel approach
utilizing audio-visual multimodal data. This method enhances audio feature
extraction by leveraging Mel Frequency Cepstral Coefficients (MFCC) and Log-Mel
spectrogram features alongside a pre-trained VGGish network. Moreover, this
paper adaptively captures fusion features across modalities by modeling the
temporal relationships, and ultilizes a pre-trained GPT-2 model for
sophisticated context-aware fusion of multimodal information. Our method
notably improves the accuracy of AU detection by understanding the temporal and
contextual nuances of the data, showcasing significant advancements in the
comprehension of intricate scenarios. These findings underscore the potential
of integrating temporal dynamics and contextual interpretation, paving the way
for future research endeavors.

利用音频数据和视觉数据协同作用对于理解人类情绪和行为非常重要，本论文提出了一种利用音视频多模态数据的新方法，通过模型化时间关系和利用预训练的 GPT-2 模型进行上下文感知的多模态信息融合，显著提高了面部动作单元检测的准确性，突显了对复杂场景理解的重要进展，为未来研究铺平了道路。