Motivated by the success of T5 (Text-To-Text Transfer Transformer) in
pre-trained natural language processing models, we propose a unified-modal
SpeechT5 framework that explores the encoder-decoder pre-training for
self-supervised speech/text representation learning. The SpeechT5 framework
consists of a shared encoder-decoder network and six modal-specific
(speech/text) pre/post-nets. After preprocessing the input speech/text through
the pre-nets, the shared encoder-decoder network models the
sequence-to-sequence transformation, and then the post-nets generate the output
in the speech/text modality based on the output of the decoder. Leveraging
large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a
unified-modal representation, hoping to improve the modeling capability for
both speech and text. To align the textual and speech information into this
unified semantic space, we propose a cross-modal vector quantization approach
that randomly mixes up speech/text states with latent units as the interface
between encoder and decoder. Extensive evaluations show the superiority of the
proposed SpeechT5 framework on a wide variety of spoken language processing
tasks, including automatic speech recognition, speech synthesis, speech
translation, voice conversion, speech enhancement, and speaker identification.
We release our code and model at this https URL

本文介绍了一种名为 SpeechT5 的统一多模态语音文本表示学习框架，该框架采用编码器 - 解码器预训练的方法进行自监督学习，具有包括自动语音识别、语音合成、语音翻译、语音增强和说话人识别在内的广泛的口语处理任务上的卓越性能。