Measuring automatic speech recognition (ASR) system quality is critical for
creating user-satisfying voice-driven applications. Word Error Rate (WER) has
been traditionally used to evaluate ASR system quality; however, it sometimes
correlates poorly with user perception/judgement of transcription quality. This
is because WER weighs every word equally and does not consider semantic
correctness which has a higher impact on user perception. In this work, we
propose evaluating ASR output hypotheses quality with SemDist that can measure
semantic correctness by using the distance between the semantic vectors of the
reference and hypothesis extracted from a pre-trained language model. Our
experimental results of 71K and 36K user annotated ASR output quality show that
SemDist achieves higher correlation with user perception than WER. We also show
that SemDist has higher correlation with downstream Natural Language
Understanding (NLU) tasks than WER.

本研究提出使用使用预训练语言模型提取参考和假设的语义向量之间的距离来衡量 ASR 输出假设的语义正确性，实验结果表明，SemDist 比 WER 更能与用户感知相一致，并且比 WER 更能与下游的自然语言理解任务相一致。