We present a joint Speech and Language Model (SLM), a multitask,
multilingual, and dual-modal model that takes advantage of pretrained
foundational speech and language models. SLM freezes the pretrained foundation
models to maximally preserves their capabilities, and only trains a simple
adapter with just 1\% (156M) of the foundation models' parameters. This
adaptation not only leads SLM to achieve strong performance on conventional
tasks such as speech recognition (ASR) and speech translation (AST), but also
introduces the novel capability of zero-shot instruction-following for more
diverse tasks: given a speech input and a text instruction, SLM is able to
perform unseen generation tasks including contextual biasing ASR using
real-time context, dialog generation, speech continuation, and question
answering, etc. Our approach demonstrates that the representational gap between
pretrained speech and language models might be narrower than one would expect,
and can be bridged by a simple adaptation mechanism. As a result, SLM is not
only efficient to train, but also inherits strong capabilities already acquired
in foundation models of different modalities.

我们提出了一种联合语音与语言模型（SLM），它是一种多任务、多语种、双模态的模型，充分利用了预训练的语音和语言基础模型。SLM 通过将预训练的基础模型冻结，最大限度地保留它们的能力，并只训练一个只包含 1％（156M）基础模型参数的简单适配器，从而在传统任务（如语音识别和语音翻译）上取得了强大的性能，同时还具备了零 - shot 指导的新颖能力，能够完成包括上下文偏置语音识别、对话生成、语音延续和问答等多样化任务。我们的方法表明，预训练的语音和语言模型之间的表征差距可能比人们预期的要小，并可以通过简单的适应机制来弥合。因此，SLM 不仅训练高效，而且继承了不同模态基础模型已经具备的强大能力。