Pseudo-labeling is the most adopted method for pre-training automatic speech
recognition (ASR) models. However, its performance suffers from the supervised
teacher model's degrading quality in low-resource setups and under domain
transfer. Inspired by the successes of contrastive representation learning for
computer vision and speech applications, and more recently for supervised
learning of visual objects, we propose Contrastive Semi-supervised Learning
(CSL). CSL eschews directly predicting teacher-generated pseudo-labels in favor
of utilizing them to select positive and negative examples. In the challenging
task of transcribing public social media videos, using CSL reduces the WER by
8% compared to the standard Cross-Entropy pseudo-labeling (CE-PL) when 10hr of
supervised data is used to annotate 75,000hr of videos. The WER reduction jumps
to 19% under the ultra low-resource condition of using 1hr labels for teacher
supervision. CSL generalizes much better in out-of-domain conditions, showing
up to 17% WER reduction compared to the best CE-PL pre-trained model.

本文提出了一种名为对比半监督学习（Contrastive Semi-supervised Learning, CSL）的方法，通过挑选正负样本来替代直接预测教师生成的伪标签，并将其应用于翻译公共社交媒体视频的挑战任务中，从而比标准的交叉熵伪标签（CE-PL）减少了 8% 的 WER。