New-age conversational agent systems perform both speech emotion recognition
(SER) and automatic speech recognition (ASR) using two separate and often
independent approaches for real-world application in noisy environments. In
this paper, we investigate a joint ASR-SER multitask learning approach in a
low-resource setting and show that improvements are observed not only in SER,
but also in ASR. We also investigate the robustness of such jointly trained
models to the presence of background noise, babble, and music. Experimental
results on the IEMOCAP dataset show that joint learning can improve ASR word
error rate (WER) and SER classification accuracy by 10.7% and 2.3% respectively
in clean scenarios. In noisy scenarios, results on data augmented with MUSAN
show that the joint approach outperforms the independent ASR and SER approaches
across many noisy conditions. Overall, the joint ASR-SER approach yielded more
noise-resistant models than the independent ASR and SER approaches.

本文研究了一种联合的 ASR-SER 多任务学习方法，发现在低资源环境下，采用这种方式可通过对背景噪音的处理从而提高 ASR 词错误率和 SER 分类准确性，结果表明使得该方法产生比独立 ASR 和 SER 方法更具噪音抗干扰性的模型。