Fusing multiple modalities for affective computing tasks has proven effective
for performance improvement. However, how multimodal fusion works is not well
understood, and its use in the real world usually results in large model sizes.
In this work, on sentiment and emotion analysis, we first analyze how the
salient affective information in one modality can be affected by the other in
crossmodal attention. We find that inter-modal incongruity exists at the latent
level due to crossmodal attention. Based on this finding, we propose a
lightweight model via Hierarchical Crossmodal Transformer with Modality Gating
(HCT-MG), which determines a primary modality according to its contribution to
the target task and then hierarchically incorporates auxiliary modalities to
alleviate inter-modal incongruity and reduce information redundancy. The
experimental evaluation on three benchmark datasets: CMU-MOSI, CMU-MOSEI, and
IEMOCAP verifies the efficacy of our approach, showing that it: 1) outperforms
major prior work by achieving competitive results and can successfully
recognize hard samples; 2) mitigates the inter-modal incongruity at the latent
level when modalities have mismatched affective tendencies; 3) reduces model
size to less than 1M parameters while outperforming existing models of similar
sizes.

通过层级交叉模态变压器与模态门控的轻量级模型，本文在情感和情绪分析领域，解决了多模态数据融合存在的潜在问题，提出了一种确定主要模态并逐层整合辅助模态的方法，发现了交叉模态关注中的跨模态不一致性并在三个基准数据集上验证了该方法的有效性，并使模型参数数量减少到不到 100 万个，同时表现出具有竞争力的结果。

交叉注意力不足：不协调感知的多模态情感分析和情绪识别

Cross-Attention is Not Enough: Incongruity-Aware Multimodal Sentiment  Analysis and Emotion Recognition

Human language is often multimodal, which comprehends a mixture of natural
language, facial gestures, and acoustic behaviors. However, two major
challenges in modeling such multimodal human language time-series data exist:
1) inherent data non-alignment due to variable sampling rates for the sequences
from each modality; and 2) long-range dependencies between elements across
modalities. In this paper, we introduce the Multimodal Transformer (MulT) to
generically address the above issues in an end-to-end manner without explicitly
aligning the data. At the heart of our model is the directional pairwise
crossmodal attention, which attends to interactions between multimodal
sequences across distinct time steps and latently adapt streams from one
modality to another. Comprehensive experiments on both aligned and non-aligned
multimodal time-series show that our model outperforms state-of-the-art methods
by a large margin. In addition, empirical analysis suggests that correlated
crossmodal signals are able to be captured by the proposed crossmodal attention
mechanism in MulT.

该研究介绍了一种称为 MulT 的多模态转换器，利用方向性一对一跨模态注意力在不显式对齐数据的情况下，通用地解决了模型多模态人类语言时序数据中的两个主要挑战，即内在数据非对准性和跨模态元素之间的长程依赖关系。