We present a joint Speech and Language Model (SLM), a multitask,
multilingual, and dual-modal model that takes advantage of pretrained
foundational speech and language models. SLM freezes the pretrained foundation
models to maximally preserves their capabilities, and only trains a simple
adapter with just 1\% (156M) of the foundation models' parameters. This
adaptation not only leads SLM to achieve strong performance on conventional
tasks such as speech recognition (ASR) and speech translation (AST), but also
introduces the novel capability of zero-shot instruction-following for more
diverse tasks: given a speech input and a text instruction, SLM is able to
perform unseen generation tasks including contextual biasing ASR using
real-time context, dialog generation, speech continuation, and question
answering, etc. Our approach demonstrates that the representational gap between
pretrained speech and language models might be narrower than one would expect,
and can be bridged by a simple adaptation mechanism. As a result, SLM is not
only efficient to train, but also inherits strong capabilities already acquired
in foundation models of different modalities.

我们提出了一种联合语音与语言模型（SLM），它是一种多任务、多语种、双模态的模型，充分利用了预训练的语音和语言基础模型。SLM 通过将预训练的基础模型冻结，最大限度地保留它们的能力，并只训练一个只包含 1％（156M）基础模型参数的简单适配器，从而在传统任务（如语音识别和语音翻译）上取得了强大的性能，同时还具备了零 - shot 指导的新颖能力，能够完成包括上下文偏置语音识别、对话生成、语音延续和问答等多样化任务。我们的方法表明，预训练的语音和语言模型之间的表征差距可能比人们预期的要小，并可以通过简单的适应机制来弥合。因此，SLM 不仅训练高效，而且继承了不同模态基础模型已经具备的强大能力。

SLM：填补语音与文本基础模型之间的差距

SLM: Bridge the thin gap between speech and text foundation models

Large Language Models (LLMs) have been applied in the speech domain, often
incurring a performance drop due to misaligned between speech and language
representations. To bridge this gap, we propose a joint speech and language
model (SLM) using a Speech2Text adapter, which maps speech into text token
embedding space without speech information loss. Additionally, using a
CTC-based blank-filtering, we can reduce the speech sequence length to that of
text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the
dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to
address errors on rare entities, we augment SLM with a Speech2Entity retriever,
which uses speech to retrieve relevant entities, and then adds them to the
original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the
DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with
the dialog understanding task improves the ASR performance from 9.4% to 8.5%
WER.

本文介绍了一种使用 Speech2Text 适配器的联合语音和语言模型，用于桥接语音和语言表示之间的差距，并改进对话状态跟踪性能（DST），并通过 Speech2Entity 检索器增强了 SLM 的性能。

语音转文本适配器和语音到实体检索器增强的 LLMs 用于语音理解

Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for  Speech Understanding

We present mSLAM, a multilingual Speech and LAnguage Model that learns
cross-lingual cross-modal representations of speech and text by pre-training
jointly on large amounts of unlabeled speech and text in multiple languages.
mSLAM combines w2v-BERT pre-training on speech with SpanBERT pre-training on
character-level text, along with Connectionist Temporal Classification (CTC)
losses on paired speech and transcript data, to learn a single model capable of
learning from and representing both speech and text signals in a shared
representation space. We evaluate mSLAM on several downstream speech
understanding tasks and find that joint pre-training with text improves quality
on speech translation, speech intent classification and speech language-ID
while being competitive on multilingual ASR, when compared against speech-only
pre-training. Our speech translation model demonstrates zero-shot text
translation without seeing any text translation data, providing evidence for
cross-modal alignment of representations. mSLAM also benefits from multi-modal
fine-tuning, further improving the quality of speech translation by directly
leveraging text translation data during the fine-tuning process. Our empirical
analysis highlights several opportunities and challenges arising from
large-scale multimodal pre-training, suggesting directions for future research.

mSLAM 是一种多语言语音和语言模型，通过在多种语言的大量无标记语音和文本上联合预训练学习跨语言跨模式表示。