Humans are sophisticated at reading interlocutors' emotions from multimodal
signals, such as speech contents, voice tones and facial expressions. However,
machines might struggle to understand various emotions due to the difficulty of
effectively decoding emotions from the complex interactions between multimodal
signals. In this paper, we propose a multimodal emotion analysis framework,
InterMulti, to capture complex multimodal interactions from different views and
identify emotions from multimodal signals. Our proposed framework decomposes
signals of different modalities into three kinds of multimodal interaction
representations, including a modality-full interaction representation, a
modality-shared interaction representation, and three modality-specific
interaction representations. Additionally, to balance the contribution of
different modalities and learn a more informative latent interaction
representation, we developed a novel Text-dominated Hierarchical High-order
Fusion(THHF) module. THHF module reasonably integrates the above three kinds of
representations into a comprehensive multimodal interaction representation.
Extensive experimental results on widely used datasets, (i.e.) MOSEI, MOSI and
IEMOCAP, demonstrate that our method outperforms the state-of-the-art.

该研究提出了一种基于多模态信息的情感分析框架 ——InterMulti，并利用一种新颖的基于文本的 THHF 模块，将多种交互表现形式融合成综合的多模态交互表示来识别情感，实验结果表明，该方法的表现优于现有技术。