The prevalent approach in speech emotion recognition (SER) involves
integrating both audio and textual information to comprehensively identify the
speaker's emotion, with the text generally obtained through automatic speech
recognition (ASR). An essential issue of this approach is that ASR errors from
the text modality can worsen the performance of SER. Previous studies have
proposed using an auxiliary ASR error detection task to adaptively assign
weights of each word in ASR hypotheses. However, this approach has limited
improvement potential because it does not address the coherence of semantic
information in the text. Additionally, the inherent heterogeneity of different
modalities leads to distribution gaps between their representations, making
their fusion challenging. Therefore, in this paper, we incorporate two
auxiliary tasks, ASR error detection (AED) and ASR error correction (AEC), to
enhance the semantic coherence of ASR text, and further introduce a novel
multi-modal fusion (MF) method to learn shared representations across
modalities. We refer to our method as MF-AED-AEC. Experimental results indicate
that MF-AED-AEC significantly outperforms the baseline model by a margin of
4.1\%.

本文介绍了一种引入 ASR 错误检测和修正的辅助任务来增强 ASR 文本的语义连贯性，并进一步提出了一种新颖的多模态融合方法，名为 MF-AED-AEC。实验结果表明，MF-AED-AEC 在性能上显著优于基线模型 4.1%。