We present a novel Speech Augmented Language Model (SALM) with {\em
multitask} and {\em in-context} learning capabilities. SALM comprises a frozen
text LLM, a audio encoder, a modality adapter module, and LoRA layers to
accommodate speech input and associated task instructions. The unified SALM not
only achieves performance on par with task-specific Conformer baselines for
Automatic Speech Recognition (ASR) and Speech Translation (AST), but also
exhibits zero-shot in-context learning capabilities, demonstrated through
keyword-boosting task for ASR and AST. Moreover, {\em speech supervised
in-context training} is proposed to bridge the gap between LLM training and
downstream speech tasks, which further boosts the in-context learning ability
of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.

我们提出了一种新颖的具有多任务和上下文学习能力的语音增强语言模型（SALM）。SALM 由一个固定的文本语言模型，一个音频编码器，一个模态适配器模块和适应语音输入和关联任务指令的 LoRA 层组成。统一的 SALM 不仅在自动语音识别（ASR）和语音翻译（AST）等特定任务上达到了与 Conformer 基线相当的性能，还展现了零 - shot 的上下文学习能力，通过 ASR 和 AST 的关键词增强任务进行了验证。此外，我们提出了语音监督上下文训练方法，弥合了 LLM 训练和下游语音任务之间的差距，进一步提升了语音到文本模型的上下文学习能力。所提出的模型通过 NeMo 工具包开源。

SALM: 基于语境学习的语音增强语言模型用于语音识别和翻译

SALM: Speech-augmented Language Model with In-context Learning for  Speech Recognition and Translation

This paper presents a framework based on Weighted Finite-State Transducers
(WFST) to simplify the development of modifications for RNN-Transducer (RNN-T)
loss. Existing implementations of RNN-T use CUDA-related code, which is hard to
extend and debug. WFSTs are easy to construct and extend, and allow debugging
through visualization. We introduce two WFST-powered RNN-T implementations: (1)
"Compose-Transducer", based on a composition of the WFST graphs from acoustic
and textual schema -- computationally competitive and easy to modify; (2)
"Grid-Transducer", which constructs the lattice directly for further
computations -- most compact, and computationally efficient. We illustrate the
ease of extensibility through introduction of a new W-Transducer loss -- the
adaptation of the Connectionist Temporal Classification with Wild Cards.
W-Transducer (W-RNNT) consistently outperforms the standard RNN-T in a
weakly-supervised data setup with missing parts of transcriptions at the
beginning and end of utterances. All RNN-T losses are implemented with the k2
framework and are available in the NeMo toolkit.

本文提出了一种基于 WFST 框架的 RNN-T 损失简化开发的方案，并介绍了两种基于 WFST 的 RNN-T 实现，通过引入 W-RNNT 损失的示例说明了可扩展性优势，最终展示了 NeMo 工具包中所有实现的 RNN-T 损失。