The emotion detection technology to enhance human decision-making is an
important research issue for real-world applications, but real-life emotion
datasets are relatively rare and small. The experiments conducted in this paper
use the CEMO, which was collected in a French emergency call center. Two
pre-trained models based on speech and text were fine-tuned for speech emotion
recognition. Using pre-trained Transformer encoders mitigates our data's
limited and sparse nature. This paper explores the different fusion strategies
of these modality-specific models. In particular, fusions with and without
cross-attention mechanisms were tested to gather the most relevant information
from both the speech and text encoders. We show that multimodal fusion brings
an absolute gain of 4-9% with respect to either single modality and that the
Symmetric multi-headed cross-attention mechanism performed better than late
classical fusion approaches. Our experiments also suggest that for the
real-life CEMO corpus, the audio component encodes more emotive information
than the textual one.

本文探讨了基于 CEMO 数据集的语音和文本预训练模型在情感识别上的融合策略，研究表明基于多模态融合的交叉关注机制实现了更好的表现。同时，实验也表明在 CEMO 上，音频编码包含的情感信息比文本更为丰富。