We describe a method to jointly pre-train speech and text in an
encoder-decoder modeling framework for speech translation and recognition. The
proposed method incorporates four self-supervised and supervised subtasks for
cross modality learning. A self-supervised speech subtask leverages unlabelled
speech data, and a (self-)supervised text to text subtask makes use of abundant
text training data. Two auxiliary supervised speech tasks are included to unify
speech and text modeling space. Our contribution lies in integrating linguistic
information from the text corpus into the speech pre-training. Detailed
analysis reveals learning interference among subtasks. Two pre-training
configurations for speech translation and recognition, respectively, are
presented to alleviate subtask interference. Our experiments show the proposed
method can effectively fuse speech and text information into one model. It
achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the
MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the
Librispeech speech recognition task.

本文描述了一种使用编码器 - 解码器建模框架来联合预训练语音和文本用于语音翻译和识别的方法，其中包括四个自监督和有监督的子任务以进行跨模态学习，其贡献在于将文本语料库中的语言信息整合到语音预训练中。