Self-supervised learning of speech representations has been a very active
research area but most work is focused on a single domain such as read audio
books for which there exist large quantities of labeled and unlabeled data. In
this paper, we explore more general setups where the domain of the unlabeled
data for pre-training data differs from the domain of the labeled data for
fine-tuning, which in turn may differ from the test data domain. Our
experiments show that using target domain data during pre-training leads to
large performance improvements across a variety of setups. On a large-scale
competitive setup, we show that pre-training on unlabeled in-domain data
reduces the gap between models trained on in-domain and out-of-domain labeled
data by 66%-73%. This has obvious practical implications since it is much
easier to obtain unlabeled target domain data than labeled data. Moreover, we
find that pre-training on multiple domains improves generalization performance
on domains not seen during training. Code and models will be made available at
this https URL

本文探讨了语音表示的自监督学习，其中更富含挑战的是那些预训练数据的领域与微调和测试数据的领域不同的情形，试验结果表明在预训练过程中加入目标领域的数据可以显著提高性能。