Speech emotion recognition (SER) has received a great deal of attention in
recent years in the context of spontaneous conversations. While there have been
notable results on datasets like the well known corpus of naturalistic dyadic
conversations, IEMOCAP, for both the case of categorical and dimensional
emotions, there are few papers which try to predict both paradigms at the same
time. Therefore, in this work, we aim to highlight the performance contribution
of multi-task learning by proposing a multi-task, multi-modal system that
predicts categorical and dimensional emotions. The results emphasise the
importance of cross-regularisation between the two types of emotions. Our
approach consists of a multi-task, multi-modal architecture that uses parallel
feature refinement through self-attention for the feature of each modality. In
order to fuse the features, our model introduces a set of learnable bridge
tokens that merge the acoustic and linguistic features with the help of
cross-attention. Our experiments for categorical emotions on 10-fold validation
yield results comparable to the current state-of-the-art. In our configuration,
our multi-task approach provides better results compared to learning each
paradigm separately. On top of that, our best performing model achieves a high
result for valence compared to the previous multi-task experiments.

多任务学习在语音情绪识别中的应用与多模态系统和情绪范畴及情绪维度的预测相关，通过跨任务正则化和特征融合提高模型表现。

多任务多模态预测分类和维度情感

A Multi-Task, Multi-Modal Approach for Predicting Categorical and  Dimensional Emotions

Estimating dimensional emotions, such as activation, valence and dominance,
from acoustic speech signals has been widely explored over the past few years.
While accurate estimation of activation and dominance from speech seem to be
possible, the same for valence remains challenging. Previous research has shown
that the use of lexical information can improve valence estimation performance.
Lexical information can be obtained from pre-trained acoustic models, where the
learned representations can improve valence estimation from speech. We
investigate the use of pre-trained model representations to improve valence
estimation from acoustic speech signal. We also explore fusion of
representations to improve emotion estimation across all three emotion
dimensions: activation, valence and dominance. Additionally, we investigate if
representations from pre-trained models can be distilled into models trained
with low-level features, resulting in models with a less number of parameters.
We show that fusion of pre-trained model embeddings result in a 79% relative
improvement in concordance correlation coefficient CCC on valence estimation
compared to standard acoustic feature baseline (mel-filterbank energies), while
distillation from pre-trained model embeddings to lower-dimensional
representations yielded a relative 12% improvement. Such performance gains were
observed over two evaluation sets, indicating that our proposed architecture
generalizes across those evaluation sets. We report new state-of-the-art
"text-free" acoustic-only dimensional emotion estimation $CCC$ values on two
MSP-Podcast evaluation sets.

本研究探讨利用预训练的声学模型，将词汇信息融合到声学语音信号中，以改善情感估计，特别是情感维度中的愉悦度估计，并且发现预训练的模型嵌入融合可比标准声学特征基线（Mel 滤波器的能量）产生更好的效果，且经测试可以推广到其他数据集上。