We introduce SPIRIT-LM, a foundation multimodal language model that freely
mixes text and speech. Our model is based on a pretrained text language model
that we extend to the speech modality by continuously training it on text and
speech units. Speech and text sequences are concatenated as a single set of
tokens, and trained with a word-level interleaving method using a small
automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two
versions: a BASE version that uses speech semantic units and an EXPRESSIVE
version that models expressivity using pitch and style units in addition to the
semantic units. For both versions, the text is encoded with subword BPE tokens.
The resulting model displays both the semantic abilities of text models and the
expressive abilities of speech models. Additionally, we demonstrate that
SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities
(i.e. ASR, TTS, Speech Classification).

我们介绍了 SPIRIT-LM，这是一个混合文本与语音的基础多模态语言模型。我们基于一个预训练的文本语言模型扩展到语音模态，通过在文本和语音单元上进行连续训练来实现。语音和文本序列被连接为一组单词，并使用小型的自动化筛选的语音 - 文本平行语料库以单词级交错方法进行训练。SPIRIT-LM 有两个版本：一个基础版本使用语音语义单元，一个富有表现力版本除了语义单元，还使用音调和风格单元进行表达能力建模。对于两个版本，文本使用子词 BPE 单元进行编码。该模型展示了文本模型的语义能力和语音模型的表达能力。此外，我们证明 SPIRIT-LM 能够跨多模态（如：ASR，TTS，语音分类）以几次学习的方式学习新任务。