We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.

使用预先训练、自我训练和模型规模扩大的方法，利用大型、多样化的未标记数据集，我们取得了一系列使用巨型自动语音识别（ASR）模型的结果，表明它可以在极大的任务效率下使用仅有少量标记数据来实现最先进性能。在一个含有34k个小时标记数据的ASR任务中，通过微调一个80亿参数的预训练Conformer模型，我们可以实现仅有3％的训练数据即可达到SoTA性能，并且使用完整的训练集可以显著提高SoTA。我们还报告了使用大型预训练和自我训练模型对多个下游任务的通用收益，涵盖了许多语音领域的公共基准测试，并跨越了多个数据集大小的数量级，并利用预训练网络的学习表示在非ASR任务中取得了SoTA结果。

BigSSL：大规模半监督学习在自动语音识别中的探索前沿