Speech language models (SpeechLMs) process and generate acoustic data only,
without textual supervision. In this work, we propose TWIST, a method for
training SpeechLMs using a warm-start from a pretrained textual language
models. We show using both automatic and human evaluations that TWIST
outperforms a cold-start SpeechLM across the board. We empirically analyze the
effect of different model design choices such as the speech tokenizer, the
pretrained textual model, and the dataset size. We find that model and dataset
scale both play an important role in constructing better-performing SpeechLMs.
Based on our observations, we present the largest (to the best of our
knowledge) SpeechLM both in terms of number of parameters and training data. We
additionally introduce two spoken versions of the StoryCloze textual benchmark
to further improve model evaluation and advance future research in the field.
Speech samples can be found on our website:
this https URL .

该研究提出了 TWIST 方法，利用预训练的文本语言模型进行暖启动，从而训练语音语言模型，实验结果表明 TWIST 方法在多个方面胜过冷启动方法。基于观察结果，研究团队提出了迄今为止最大的 SpeechLM，并引入了两个口语版的 StoryCloze 文本基准来进一步改进模型评估和推动未来研究进展。