The prevalent approach in speech emotion recognition (SER) involves
integrating both audio and textual information to comprehensively identify the
speaker's emotion, with the text generally obtained through automatic speech
recognition (ASR). An essential issue of this approach is that ASR errors from
the text modality can worsen the performance of SER. Previous studies have
proposed using an auxiliary ASR error detection task to adaptively assign
weights of each word in ASR hypotheses. However, this approach has limited
improvement potential because it does not address the coherence of semantic
information in the text. Additionally, the inherent heterogeneity of different
modalities leads to distribution gaps between their representations, making
their fusion challenging. Therefore, in this paper, we incorporate two
auxiliary tasks, ASR error detection (AED) and ASR error correction (AEC), to
enhance the semantic coherence of ASR text, and further introduce a novel
multi-modal fusion (MF) method to learn shared representations across
modalities. We refer to our method as MF-AED-AEC. Experimental results indicate
that MF-AED-AEC significantly outperforms the baseline model by a margin of
4.1\%.

本文介绍了一种引入 ASR 错误检测和修正的辅助任务来增强 ASR 文本的语义连贯性，并进一步提出了一种新颖的多模态融合方法，名为 MF-AED-AEC。实验结果表明，MF-AED-AEC 在性能上显著优于基线模型 4.1%。

MF-AED-AEC: 利用多模态融合、ASR 错误检测和 ASR 错误修正进行语音情感识别

MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion,  ASR Error Detection, and ASR Error Correction

ASR Error Detection (AED) models aim to post-process the output of Automatic
Speech Recognition (ASR) systems, in order to detect transcription errors.
Modern approaches usually use text-based input, comprised solely of the ASR
transcription hypothesis, disregarding additional signals from the ASR model.
Instead, we propose to utilize the ASR system's word-level confidence scores
for improving AED performance. Specifically, we add an ASR Confidence Embedding
(ACE) layer to the AED model's encoder, allowing us to jointly encode the
confidence scores and the transcribed text into a contextualized
representation. Our experiments show the benefits of ASR confidence scores for
AED, their complementary effect over the textual signal, as well as the
effectiveness and robustness of ACE for combining these signals. To foster
further research, we publish a novel AED dataset consisting of ASR outputs on
the LibriSpeech corpus with annotated transcription errors.

该研究提出了一种基于自动语音识别 (ASR) 系统的置信得分，改善 ASR 误差检测 (AED) 模型的性能，并证明了这种置信得分的补充作用以及 ASR Confidence Embedding 的有效性和鲁棒性，并发布了一个包含有标记的 LibriSpeech 语料库的 ASR 输出的新型 AED 数据集。