Word error rate (WER) is a standard metric for the evaluation of Automated
Speech Recognition (ASR) systems. However, WER fails to provide a fair
evaluation of human perceived quality in presence of spelling variations,
abbreviations, or compound words arising out of agglutination. Multiple
spelling variations might be acceptable based on locale/geography, alternative
abbreviations, borrowed words, and transliteration of code-mixed words from a
foreign language to the target language script. Similarly, in case of
agglutination, often times the agglutinated, as well as the split forms, are
acceptable. Previous work handled this problem by using manually identified
normalization pairs and applying them to both the transcription and the
hypothesis before computing WER. In this paper, we propose an automatic WER
normalization system consisting of two modules: spelling normalization and
segmentation normalization. The proposed system is unsupervised and language
agnostic, and therefore scalable. Experiments with ASR on 35K utterances across
four languages yielded an average WER reduction of 13.28%. Human judgements of
these automatically identified normalization pairs show that our WER-normalized
evaluation is highly consistent with the perceived quality of ASR output.

提出了一种基于拼写和分词的自动化 WER 规范系统，通过在四种语言上的实验，平均 WER 降低了 13.28％。同时，基于该系统的标注数据的人工实验结果表明，该文中所提出的 WER 规范方法与 ASR 输出的感知质量高度一致。