Unpaired data has shown to be beneficial for low-resource automatic speech
recognition~(ASR), which can be involved in the design of hybrid models with
multi-task training or language model dependent pre-training. In this work, we
leverage unpaired data to train a general sequence-to-sequence model. Unpaired
speech and text are used in the form of data pairs by generating the
corresponding missing parts in prior to model training. Inspired by the
complementarity of speech-PseudoLabel pair and SynthesizedAudio-text pair in
both acoustic features and linguistic features, we propose a complementary
joint training~(CJT) method that trains a model alternatively with two data
pairs. Furthermore, label masking for pseudo-labels and gradient restriction
for synthesized audio are proposed to further cope with the deviations from
real data, termed as CJT++. Experimental results show that compared to
speech-only training, the proposed basic CJT achieves great performance
improvements on clean/other test sets, and the CJT++ re-training yields further
performance enhancements. It is also apparent that the proposed method
outperforms the wav2vec2.0 model with the same model size and beam size,
particularly in extreme low-resource cases.

本篇论文介绍了如何利用未配对的语音和文本数据，通过生成相应的缺失部分进行模型训练，并提出了一种称为 CJT++ 的补充联合训练方法，包括伪标签的标签掩蔽和合成音频的梯度限制，以应对与真实数据的偏差。实验结果表明，相比于仅使用语音进行训练，所提出的基本 CJT 方法在干净 / 其他测试集上实现了显著的性能提升，CJT++ 重新训练进一步增强了性能，并在极低资源情况下特别优于相同模型大小和波束大小的 wav2vec2.0 模型。