Although modern automatic speech recognition (ASR) systems can achieve high
performance, they may produce errors that weaken readers' experience and do
harm to downstream tasks. To improve the accuracy and reliability of ASR
hypotheses, we propose a cross-modal post-processing system for speech
recognizers, which 1) fuses acoustic features and textual features from
different modalities, 2) joints a confidence estimator and an error corrector
in multi-task learning fashion and 3) unifies error correction and utterance
rejection modules. Compared with single-modal or single-task models, our
proposed system is proved to be more effective and efficient. Experiment result
shows that our post-processing system leads to more than 10% relative reduction
of character error rate (CER) for both single-speaker and multi-speaker speech
on our industrial ASR system, with about 1.7ms latency for each token, which
ensures that extra latency introduced by post-processing is acceptable in
streaming speech recognition.

本文提出了一种跨模态后处理系统，包括融合不同模态的声学特征和文本特征、联合置信度估计器和错误修正器以及统一的错误修正和话语拒绝模块等，证明相较于单模型或单任务模型，该系统更加有效率地减小语音识别中字符错误率 (CER)，且每个单词的额外延迟在可接受范围内。