Prompts are crucial to large language models as they provide context
information such as topic or logical relationships. Inspired by this, we
propose PromptASR, a framework that integrates prompts in end-to-end automatic
speech recognition (E2E ASR) systems to achieve contextualized ASR with
controllable style of transcriptions. Specifically, a dedicated text encoder
encodes the text prompts and the encodings are injected into the speech encoder
by cross-attending the features from two modalities. When using the ground
truth text from preceding utterances as content prompt, the proposed system
achieves 21.9% and 6.8% relative word error rate reductions on a book reading
dataset and an in-house dataset compared to a baseline ASR system. The system
can also take word-level biasing lists as prompt to improve recognition
accuracy on rare words. An additional style prompt can be given to the text
encoder and guide the ASR system to output different styles of transcriptions.
The code is available at icefall.

使用提示将上下文信息以控制风格的方式整合到端到端自动语音识别系统中，从而提高识别准确性和降低词错误率。

具有可控风格的上下文感知语音识别的 PromptASR

PromptASR for contextualized ASR with controllable style

Recently, there has been a growing interest in the field of controllable
Text-to-Speech (TTS). While previous studies have relied on users providing
specific style factor values based on acoustic knowledge or selecting reference
speeches that meet certain requirements, generating speech solely from natural
text prompts has emerged as a new challenge for researchers. This challenge
arises due to the scarcity of high-quality speech datasets with natural text
style prompt and the absence of advanced text-controllable TTS models. In light
of this, 1) we propose TextrolSpeech, which is the first large-scale speech
emotion dataset annotated with rich text attributes. The dataset comprises
236,220 pairs of style prompt in natural text descriptions with five style
factors and corresponding speech samples. Through iterative experimentation, we
introduce a multi-stage prompt programming approach that effectively utilizes
the GPT model for generating natural style descriptions in large volumes. 2)
Furthermore, to address the need for generating audio with greater style
diversity, we propose an efficient architecture called Salle. This architecture
treats text controllable TTS as a language model task, utilizing audio codec
codes as an intermediate representation to replace the conventional
mel-spectrogram. Finally, we successfully demonstrate the ability of the
proposed model by showing a comparable performance in the controllable TTS
task. Audio samples are available at this https URL

我们提出了 TextrolSpeech，这是一个与丰富文本属性一起注释的首个大规模语音情感数据集。该数据集包含 236,220 对以自然文本描述为样式提示的样式提示和对应的语音样本。此外，我们还提出了一种名为 Salle 的高效架构，将文本可控 TTS 作为一个语言模型任务处理，利用音频编解码代码作为中间表示来替代传统的 mel 频谱图。最终，我们成功展示了该模型在可控 TTS 任务中具有可比较的性能。