In this paper, we present a novel deep multimodal framework to predict human
emotions based on sentence-level spoken language. Our architecture has two
distinctive characteristics. First, it extracts the high-level features from
both text and audio via a hybrid deep multimodal structure, which considers the
spatial information from text, temporal information from audio, and high-level
associations from low-level handcrafted features. Second, we fuse all features
by using a three-layer deep neural network to learn the correlations across
modalities and train the feature extraction and fusion modules together,
allowing optimal global fine-tuning of the entire structure. We evaluated the
proposed framework on the IEMOCAP dataset. Our result shows promising
performance, achieving 60.4% in weighted accuracy for five emotion categories.

本文提出了一种新颖的深度多模态框架，基于句子级别的口语语言预测人类情感，通过混合式的深度多模态结构从文本和音频中提取高级特征，再使用三层深度神经网络将所有特征融合起来进行训练，实现整个结构的最优全局微调，结果表明本文框架在 IEMOCAP 数据集上达到了 60.4% 的加权准确率。