In the wake of the surging tide of deep learning over the past decade,
Automatic Speech Recognition (ASR) has garnered substantial attention, leading
to the emergence of numerous publicly accessible ASR systems that are actively
being integrated into our daily lives. Nonetheless, the impartial and
replicable evaluation of these ASR systems encounters challenges due to various
crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a
general-purpose, open-source platform designed for ASR evaluation. With this
platform: (i) We report a comprehensive benchmark, unveiling the current
state-of-the-art panorama for ASR systems, covering both open-source models and
industrial commercial services. (ii) We quantize how distinct nuances in the
scoring pipeline influence the final benchmark outcomes. These include nuances
related to capitalization, punctuation, interjection, contraction, synonym
usage, compound words, etc. These issues have gained prominence in the context
of the transition towards an End-to-End future. (iii) We propose a practical
modification to the conventional Token-Error-Rate (TER) evaluation metric, with
inspirations from Kolmogorov complexity and Normalized Information Distance
(NID). This adaptation, called modified-TER (mTER), achieves proper
normalization and symmetrical treatment of reference and hypothesis. By
leveraging this platform as a large-scale testing ground, this study
demonstrates the robustness and backward compatibility of mTER when compared to
TER. The SpeechColab Leaderboard is accessible at
this https URL

本研究介绍了一个通用、开源的自动语音识别评估平台 SpeechColab Leaderboard，并使用此平台报告了一项全面的基准测试，揭示了自动语音识别系统的最新发展状况，包括开源模型和商业服务，还修改了传统的 Token-Error-Rate（TER）评估指标，提出了一种名为 modified-TER（mTER）的适应性改进，该研究通过大规模测试表明 mTER 的鲁棒性和向后兼容性。