Existing works on multimodal affective computing tasks, such as emotion
recognition, generally adopt a two-phase pipeline, first extracting feature
representations for each single modality with hand-crafted algorithms and then
performing end-to-end learning with the extracted features. However, the
extracted features are fixed and cannot be further fine-tuned on different
target tasks, and manually finding feature extraction algorithms does not
generalize or scale well to different tasks, which can lead to sub-optimal
performance. In this paper, we develop a fully end-to-end model that connects
the two phases and optimizes them jointly. In addition, we restructure the
current datasets to enable the fully end-to-end training. Furthermore, to
reduce the computational overhead brought by the end-to-end model, we introduce
a sparse cross-modal attention mechanism for the feature extraction.
Experimental results show that our fully end-to-end model significantly
surpasses the current state-of-the-art models based on the two-phase pipeline.
Moreover, by adding the sparse cross-modal attention, our model can maintain
performance with around half the computation in the feature extraction part.

本研究开发了一种完全的端到端模型，用于连接两个阶段并共同优化它们，以解决情感计算任务中现有两阶段流程的性能问题。同时我们引入了稀疏交叉模态注意机制来降低计算复杂度并在实验中证实其优越性。

情感识别的多模态端到端稀疏模型

Multimodal End-to-End Sparse Model for Emotion Recognition

Multimodal affective computing, learning to recognize and interpret human
affects and subjective information from multiple data sources, is still
challenging because: (i) it is hard to extract informative features to
represent human affects from heterogeneous inputs; (ii) current fusion
strategies only fuse different modalities at abstract level, ignoring
time-dependent interactions between modalities. Addressing such issues, we
introduce a hierarchical multimodal architecture with attention and word-level
fusion to classify utter-ance-level sentiment and emotion from text and audio
data. Our introduced model outperforms the state-of-the-art approaches on
published datasets and we demonstrated that our model is able to visualize and
interpret the synchronized attention over modalities.

本文研究了从多个数据来源中学习识别和解释人类情感和主观信息的多模态情感计算问题，提出了一种基于 attention 机制和单词级融合的分层多模态体系结构，以从文本和音频数据中分类识别发言级别的情感和情绪。结果表明，该模型在现有数据集上表现优异。