This paper studies a novel pre-training technique with unpaired speech data,
Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within
a multi-task learning framework, we introduce two pre-training tasks for the
encoder-decoder network using acoustic units, i.e., pseudo codes, derived from
an offline clustering model. One is to predict the pseudo codes via masked
language modeling in encoder output, like HuBERT model, while the other lets
the decoder learn to reconstruct pseudo codes autoregressively instead of
generating textual scripts. In this way, the decoder learns to reconstruct
original speech information with codes before learning to generate correct
text. Comprehensive experiments on the LibriSpeech corpus show that the
proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over
the method without decoder pre-training, and also outperforms significantly the
state-of-the-art wav2vec 2.0 and HuBERT on fine-tuning subsets of 10h and 100h.
We release our code and model at
this https URL

本文在多任务学习框架下，提出了一种使用新型无配对语音数据的预训练技术 Speech2C，通过声学单元 (pseudo codes) 派生自离线聚类模型，提出两个预训练任务来训练 “编码器 - 解码器” 网络，其一是像 HuBERT 模型一样通过掩蔽语言模型在编码器输出中预测 pseudo codes，其二则是让解码器自回归地重构 pseudo codes。此方法能够在 LibriSpeech 语料库上相对于没有解码器预训练的方法将字词错误率 (WER) 降低 19.2%，并且在 10h 和 100h 的微调子集上显著优于最先进的 wav2vec 2.0 和 HuBERT。