This paper deals with the utterance-level modalities missing problem with uncertain patterns on emotion recognition in conversation (ERC) task. Present models generally predict the speaker's emotions by its current utterance and context, which is degraded by modality missing considerably. Our work proposes a framework Missing-Modality Robust emotion Recognition (M2R2), which trains emotion recognition model with iterative data augmentation by learned common representation. Firstly, a network called Party Attentive Network (PANet) is designed to classify emotions, which tracks all the speakers' states and context. Attention mechanism between speaker with other participants and dialogue topic is used to decentralize dependence on multi-time and multi-party utterances instead of the possible incomplete one. Moreover, the Common Representation Learning (CRL) problem is defined for modality-missing problem. Data imputation methods improved by the adversarial strategy are used here to construct extra features to augment data. Extensive experiments and case studies validate the effectiveness of our methods over baselines for modality-missing emotion recognition on two different datasets.

本文讨论了缺失情感识别对话（ERC）任务中对话级情感识别以及不确定模式下的问题。我们提出了一种基于缺失模态的鲁棒情感识别框架（M2R2），其中通过学习通用表示进行迭代数据增强来训练情感识别模型。该框架采用了Party Attentive Network（PANet）来跟踪所有发言人的状态和上下文，并通过采用与其他参与者之间的关注机制和对话主题来使对多时间和多方发言的依赖性分散化，从而解决了情感识别中的缺失问题。同时，我们还通过以对抗策略提高了缺失数据插值方法的数据增强效果，从而为缺失模态情感识别问题建立了额外特征。广泛的实验证明了我们的方法比基线更有效，适用于不同数据集的缺失模态情感识别。

M2R2: 迭代数据增强的缺失模态鲁棒情感识别框架