Multimodal emotion recognition has attracted much attention recently. Fusing
multiple modalities effectively with limited labeled data is a challenging
task. Considering the success of pre-trained model and fine-grained nature of
emotion expression, it is reasonable to take these two aspects into
consideration. Unlike previous methods that mainly focus on one aspect, we
introduce a novel multi-granularity framework, which combines fine-grained
representation with pre-trained utterance-level representation. Inspired by
Transformer TTS, we propose a multilevel transformer model to perform
fine-grained multimodal emotion recognition. Specifically, we explore different
methods to incorporate phoneme-level embedding with word-level embedding. To
perform multi-granularity learning, we simply combine multilevel transformer
model with Albert. Extensive experimental results show that both our multilevel
transformer model and multi-granularity model outperform previous
state-of-the-art approaches on IEMOCAP dataset with text transcripts and speech
signal.

本文提出一种新的多层次转换器模型，将细粒度表示和预训练的话语级别表示相结合，结合不同的方法将音素级别嵌入与单词级别嵌入相结合，从而进行细粒度的多模式情感识别，最终在 IEMOCAP 数据集上，我们的模型均优于之前的最优方法。

多模态情感识别的多级 Transformer

Multilevel Transformer For Multimodal Emotion Recognition

The research and applications of multimodal emotion recognition have become
increasingly popular recently. However, multimodal emotion recognition faces
the challenge of lack of data. To solve this problem, we propose to use
transfer learning which leverages state-of-the-art pre-trained models including
wav2vec 2.0 and BERT for this task. Multi-level fusion approaches including
coattention-based early fusion and late fusion with the models trained on both
embeddings are explored. Also, a multi-granularity framework which extracts not
only frame-level speech embeddings but also segment-level embeddings including
phone, syllable and word-level speech embeddings is proposed to further boost
the performance. By combining our coattention-based early fusion model and late
fusion model with the multi-granularity feature extraction framework, we obtain
result that outperforms best baseline approaches by 1.3% unweighted accuracy
(UA) on the IEMOCAP dataset.

该研究探讨了多模态情感识别问题，提出了使用迁移学习的方法，结合各种预训练模型和多级融合方法，以及多粒度特征提取的框架，从而在 IEMOCAP 数据集上取得了比最佳基线法高 1.3％的性能表现。