The research and applications of multimodal emotion recognition have become
increasingly popular recently. However, multimodal emotion recognition faces
the challenge of lack of data. To solve this problem, we propose to use
transfer learning which leverages state-of-the-art pre-trained models including
wav2vec 2.0 and BERT for this task. Multi-level fusion approaches including
coattention-based early fusion and late fusion with the models trained on both
embeddings are explored. Also, a multi-granularity framework which extracts not
only frame-level speech embeddings but also segment-level embeddings including
phone, syllable and word-level speech embeddings is proposed to further boost
the performance. By combining our coattention-based early fusion model and late
fusion model with the multi-granularity feature extraction framework, we obtain
result that outperforms best baseline approaches by 1.3% unweighted accuracy
(UA) on the IEMOCAP dataset.

该研究探讨了多模态情感识别问题，提出了使用迁移学习的方法，结合各种预训练模型和多级融合方法，以及多粒度特征提取的框架，从而在 IEMOCAP 数据集上取得了比最佳基线法高 1.3％的性能表现。