Existing works on multimodal affective computing tasks, such as emotion
recognition, generally adopt a two-phase pipeline, first extracting feature
representations for each single modality with hand-crafted algorithms and then
performing end-to-end learning with the extracted features. However, the
extracted features are fixed and cannot be further fine-tuned on different
target tasks, and manually finding feature extraction algorithms does not
generalize or scale well to different tasks, which can lead to sub-optimal
performance. In this paper, we develop a fully end-to-end model that connects
the two phases and optimizes them jointly. In addition, we restructure the
current datasets to enable the fully end-to-end training. Furthermore, to
reduce the computational overhead brought by the end-to-end model, we introduce
a sparse cross-modal attention mechanism for the feature extraction.
Experimental results show that our fully end-to-end model significantly
surpasses the current state-of-the-art models based on the two-phase pipeline.
Moreover, by adding the sparse cross-modal attention, our model can maintain
performance with around half the computation in the feature extraction part.

本研究开发了一种完全的端到端模型，用于连接两个阶段并共同优化它们，以解决情感计算任务中现有两阶段流程的性能问题。同时我们引入了稀疏交叉模态注意机制来降低计算复杂度并在实验中证实其优越性。