Code-switching is a widely prevalent linguistic phenomenon in multilingual
societies like India. Building speech-to-text models for code-switched speech
is challenging due to limited availability of datasets. In this work, we focus
on the problem of spoken translation (ST) of code-switched speech in Indian
languages to English text. We present a new end-to-end model architecture COSTA
that scaffolds on pretrained automatic speech recognition (ASR) and machine
translation (MT) modules (that are more widely available for many languages).
Speech and ASR text representations are fused using an aligned interleaving
scheme and are fed further as input to a pretrained MT module; the whole
pipeline is then trained end-to-end for spoken translation using synthetically
created ST data. We also release a new evaluation benchmark for code-switched
Bengali-English, Hindi-English, Marathi-English and Telugu- English speech to
English text. COSTA significantly outperforms many competitive cascaded and
end-to-end multimodal baselines by up to 3.5 BLEU points.

通过先前训练好的自动语音识别 (ASR) 和机器翻译 (MT) 模块的结合，本研究提出了一种名为 COSTA 的端到端模型架构，用于印度语言到英文文本的混合语言实时翻译，并发布出相关评估基准。COSTA 在混合语言的孟加拉语、印地语、马拉地语和泰卢固语到英文文本的翻译中，相对于其他基线模型表现出显著优势，BLEU 点数提高达 3.5。

CoSTA: 使用对齐的语音文本交替进行混合编码的语音翻译

CoSTA: Code-Switched Speech Translation using Aligned Speech-Text  Interleaving

We present a novel Speech Augmented Language Model (SALM) with {\em
multitask} and {\em in-context} learning capabilities. SALM comprises a frozen
text LLM, a audio encoder, a modality adapter module, and LoRA layers to
accommodate speech input and associated task instructions. The unified SALM not
only achieves performance on par with task-specific Conformer baselines for
Automatic Speech Recognition (ASR) and Speech Translation (AST), but also
exhibits zero-shot in-context learning capabilities, demonstrated through
keyword-boosting task for ASR and AST. Moreover, {\em speech supervised
in-context training} is proposed to bridge the gap between LLM training and
downstream speech tasks, which further boosts the in-context learning ability
of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.

我们提出了一种新颖的具有多任务和上下文学习能力的语音增强语言模型（SALM）。SALM 由一个固定的文本语言模型，一个音频编码器，一个模态适配器模块和适应语音输入和关联任务指令的 LoRA 层组成。统一的 SALM 不仅在自动语音识别（ASR）和语音翻译（AST）等特定任务上达到了与 Conformer 基线相当的性能，还展现了零 - shot 的上下文学习能力，通过 ASR 和 AST 的关键词增强任务进行了验证。此外，我们提出了语音监督上下文训练方法，弥合了 LLM 训练和下游语音任务之间的差距，进一步提升了语音到文本模型的上下文学习能力。所提出的模型通过 NeMo 工具包开源。