Humans are skilled in reading the interlocutor's emotion from multimodal
signals, including spoken words, simultaneous speech, and facial expressions.
It is still a challenge to effectively decode emotions from the complex
interactions of multimodal signals. In this paper, we design three kinds of
multimodal latent representations to refine the emotion analysis process and
capture complex multimodal interactions from different views, including a
intact three-modal integrating representation, a modality-shared
representation, and three modality-individual representations. Then, a
modality-semantic hierarchical fusion is proposed to reasonably incorporate
these representations into a comprehensive interaction representation. The
experimental results demonstrate that our EffMulti outperforms the
state-of-the-art methods. The compelling performance benefits from its
well-designed framework with ease of implementation, lower computing
complexity, and less trainable parameters.

本文提出了一种多模态情感分析框架 EffMulti，采用三种不同的多模态潜在表示来捕捉不同视角下的复杂情感交互，并提出了一种模态 - 语义层次融合方法来融合这些表示，实验结果表明，EffMulti 在进行情感分析方面表现极佳。