Commonly used speech corpora inadequately challenge academic and commercial
ASR systems. In particular, speech corpora lack metadata needed for detailed
analysis and WER measurement. In response, we present Earnings-21, a 39-hour
corpus of earnings calls containing entity-dense speech from nine different
financial sectors. This corpus is intended to benchmark ASR systems in the wild
with special attention towards named entity recognition. We benchmark four
commercial ASR models, two internal models built with open-source tools, and an
open-source LibriSpeech model and discuss their differences in performance on
Earnings-21. Using our recently released fstalign tool, we provide a candid
analysis of each model's recognition capabilities under different partitions.
Our analysis finds that ASR accuracy for certain NER categories is poor,
presenting a significant impediment to transcript comprehension and usage.
Earnings-21 bridges academic and commercial ASR system evaluation and enables
further research on entity modeling and WER on real world audio.

本文提出了一个 39 小时的包含来自不同金融领域的密集实体语音的 Earnings-21 语料库，用于 ASR 系统的基准测试，并针对命名实体识别进行特殊关注。通过我们最近发布的 fstalign 工具，提供了每个模型在不同划分下的识别能力的坦率分析。分析发现，某些 NER 类别的 ASR 准确率很低，存在着显著的文本理解和使用障碍。Earnings-21 能够评估学术和商业 ASR 系统，使得实体塑造和 WER 在真实世界音频上的进一步研究成为可能。