Self-supervised learning via masked prediction pre-training (MPPT) has shown
impressive performance on a range of speech-processing tasks. This paper
proposes a method to bias self-supervised learning towards a specific task. The
core idea is to slightly finetune the model that is used to obtain the target
sequence. This leads to better performance and a substantial increase in
training speed. Furthermore, this paper proposes a variant of MPPT that allows
low-footprint streaming models to be trained effectively by computing the MPPT
loss on masked and unmasked frames. These approaches are evaluated for
automatic speech recognition on the Librispeech corpus, where 100 hours of data
served as the labelled data and 860 hours as the unlabelled data. The biased
training outperforms the unbiased training by 15.5% after 250k updates and
23.8% after 100k updates on test-other. For the streaming models, the
pre-training approach yields a reduction in word error rate of 44.1%.

本文提出了通过微调模型来引导自监督方法学习特定任务的方法，并提出了 MPPT 的变体，可以用于低存储占用的流式模型。这些方法在 Librispeech 语料库上进行的自动语音识别实验中取得了显著的性能提高。

自我偏置的自监督学习用于 ASR

Biased Self-supervised learning for ASR

Recently, masked prediction pre-training has seen remarkable progress in
self-supervised learning (SSL) for speech recognition. It usually requires a
codebook obtained in an unsupervised way, making it less accurate and difficult
to interpret. We propose two supervision-guided codebook generation approaches
to improve automatic speech recognition (ASR) performance and also the
pre-training efficiency, either through decoding with a hybrid ASR system to
generate phoneme-level alignments (named PBERT), or performing clustering on
the supervised speech features extracted from an end-to-end CTC model (named
CTC clustering). Both the hybrid and CTC models are trained on the same small
amount of labeled speech as used in fine-tuning. Experiments demonstrate
significant superiority of our methods to various SSL and self-training
baselines, with up to 17.0% relative WER reduction. Our pre-trained models also
show good transferability in a non-ASR speech task.

本论文提出了两种监督引导的码本生成方法，分别是使用混合 ASR 系统解码并生成音素级别对齐（命名为 PBERT）或者使用从端到端 CTC 模型中提取的受监督语音特征进行聚类（命名为 CTC 聚类），以提高自动语音识别性能和预训练效率。实验结果表明，我们的方法在各种 SSL 和自训练基线中具有显著的优越性，最高 WER 相对降低了 17.0％。我们的预训练模型在非 ASR 语音任务中也表现出良好的可迁移性。