In this paper, we propose a unified pre-training approach called UniSpeech to
learn speech representations with both unlabeled and labeled data, in which
supervised phonetic CTC learning and phonetically-aware contrastive
self-supervised learning are conducted in a multi-task learning manner. The
resultant representations can capture information more correlated with phonetic
structures and improve the generalization across languages and domains. We
evaluate the effectiveness of UniSpeech for cross-lingual representation
learning on public CommonVoice corpus. The results show that UniSpeech
outperforms self-supervised pretraining and supervised transfer learning for
speech recognition by a maximum of 13.4% and 17.8% relative phone error rate
reductions respectively (averaged over all testing languages). The
transferability of UniSpeech is also demonstrated on a domain-shift speech
recognition task, i.e., a relative word error rate reduction of 6% against the
previous approach.

本文提出了一种名为 UniSpeech 的统一预训练方法，利用有标签和无标签数据学习语音表示，其中针对语音结构的有监督 CTC 学习和语音感知对比自监督学习以多任务学习的方式进行。实验结果表明，UniSpeech 在跨语言表征学习方面的有效性超过了自监督预训练和有监督转移学习。