The common standard for quality evaluation of automatic speech recognition
(ASR) systems is reference-based metrics such as the Word Error Rate (WER),
computed using manual ground-truth transcriptions that are time-consuming and
expensive to obtain. This work proposes a multi-language referenceless quality
metric, which allows comparing the performance of different ASR models on a
speech dataset without ground truth transcriptions. To estimate the quality of
ASR hypotheses, a pre-trained language model (LM) is fine-tuned with
contrastive learning in a self-supervised learning manner. In experiments
conducted on several unseen test datasets consisting of outputs from top
commercial ASR engines in various languages, the proposed referenceless metric
obtains a much higher correlation with WER scores and their ranks than the
perplexity metric from the state-of-art multi-lingual LM in all experiments,
and also reduces WER by more than $7\%$ when used for ensembling hypotheses.
The fine-tuned model and experiments are made available for the
reproducibility: this https URL

本文提出了一种多语言无参考质量度量方法，使用预先训练的语言模型采用对比学习的自我监督学习方式进行微调，针对自动语音识别模型提出了一个评估标准，该评估标准可以在没有参考标记的情况下比较不同自动语音识别模型的性能，并在减少识别错误和提高性能方面效果显著。

基于多语言自监督对比学习的自动语音识别无参考质量度量

A Reference-less Quality Metric for Automatic Speech Recognition via  Contrastive-Learning of a Multi-Language Model with Self-Supervision

This paper introduces NoRefER, a novel referenceless quality metric for
automatic speech recognition (ASR) systems. Traditional reference-based metrics
for evaluating ASR systems require costly ground-truth transcripts. NoRefER
overcomes this limitation by fine-tuning a multilingual language model for
pair-wise ranking ASR hypotheses using contrastive learning with Siamese
network architecture. The self-supervised NoRefER exploits the known quality
relationships between hypotheses from multiple compression levels of an ASR for
learning to rank intra-sample hypotheses by quality, which is essential for
model comparisons. The semi-supervised version also uses a referenced dataset
to improve its inter-sample quality ranking, which is crucial for selecting
potentially erroneous samples. The results indicate that NoRefER correlates
highly with reference-based metrics and their intra-sample ranks, indicating a
high potential for referenceless ASR evaluation or a/b testing.

本文介绍了一种名为 NoRefER 的新型无参考质量评估指标，其采用对比学习和连体网络结构，自我监督地精调跨语言语言模型以进行自动语音识别假设的点对排序以评估质量。半监督版本还使用参考数据集来改善对潜在错误样本的选择。实验结果表明，NoRefER 与基于参考的指标高度相关，具有不错的无参考 ASR 评估或 A / B 测试潜力。