How to boost speech pre-training with textual data is an unsolved problem due
to the fact that speech and text are very different modalities with distinct
characteristics. In this paper, we propose a cross-modal Speech and Language
Model (SpeechLM) to explicitly align speech and text pre-training with a
pre-defined unified discrete representation. Specifically, we introduce two
alternative discrete tokenizers to bridge the speech and text modalities,
including phoneme-unit and hidden-unit tokenizers, which can be trained using a
small amount of paired speech-text data. Based on the trained tokenizers, we
convert the unlabeled speech and text data into tokens of phoneme units or
hidden units. The pre-training objective is designed to unify the speech and
the text into the same discrete semantic space with a unified Transformer
network. Leveraging only 10K text sentences, our SpeechLM gets a 16\% relative
WER reduction over the best base model performance (from 6.8 to 5.7) on the
public LibriSpeech ASR benchmark. Moreover, SpeechLM with fewer parameters even
outperforms previous SOTA models on CoVoST-2 speech translation tasks. We also
evaluate our SpeechLM on various spoken language processing tasks under the
universal representation evaluation framework SUPERB, demonstrating significant
improvements on content-related tasks. Our code and models are available at
this https URL.

提出了一种跨模态的语音和语言模型，使用两种不同的离散标记器来处理语音和文本模态，取得了优异的成绩，尤其表现出色的是在 CoVoST-2 语音翻译任务上。