Neural contextual biasing effectively improves automatic speech recognition
(ASR) for crucial phrases within a speaker's context, particularly those that
are infrequent in the training data. This work proposes contextual text
injection (CTI) to enhance contextual ASR. CTI leverages not only the paired
speech-text data, but also a much larger corpus of unpaired text to optimize
the ASR model and its biasing component. Unpaired text is converted into
speech-like representations and used to guide the model's attention towards
relevant bias phrases. Moreover, we introduce a contextual text-injected (CTI)
minimum word error rate (MWER) training, which minimizes the expected WER
caused by contextual biasing when unpaired text is injected into the model.
Experiments show that CTI with 100 billion text sentences can achieve up to
43.3% relative WER reduction from a strong neural biasing model. CTI-MWER
provides a further relative improvement of 23.5%.

通过上下文中注入文本的方式，使用大规模未配对的文本来优化自动语音识别（ASR）模型及其偏置成分，提高重要短语的识别准确率。实验证明，使用 1000 亿个文本句子进行 CTI，相对于强大的神经偏置模型，可以实现高达 43.3% 的相对词错误率的降低。CTI-MWER 进一步提供了 23.5% 的相对改进。

神经背景性偏倚的文本插入

Text Injection for Neural Contextual Biasing

Spoken semantic parsing (SSP) involves generating machine-comprehensible
parses from input speech. Training robust models for existing application
domains represented in training data or extending to new domains requires
corresponding triplets of speech-transcript-semantic parse data, which is
expensive to obtain. In this paper, we address this challenge by examining
methods that can use transcript-semantic parse data (unpaired text) without
corresponding speech. First, when unpaired text is drawn from existing textual
corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways
to generate speech representations for unpaired text. Experiments on the STOP
dataset show that unpaired text from existing and new domains improves
performance by 2% and 30% in absolute Exact Match (EM) respectively. Second, we
consider the setting when unpaired text is not available in existing textual
corpora. We propose to prompt Large Language Models (LLMs) to generate unpaired
text for existing and new domains. Experiments show that examples and words
that co-occur with intents can be used to generate unpaired text with Llama
2.0. Using the generated text with JAT and TTS for spoken semantic parsing
improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains
respectively.

使用现有和新领域的非配对文本可以提高已有和新领域的口语语义解析的性能，同时，通过提示大型语言模型生成非配对文本，可以进一步提升性能。

使用大型语言模型增强口语理解的文本

Augmenting text for spoken language understanding with Large Language  Models

Unpaired text and audio injection have emerged as dominant methods for
improving ASR performance in the absence of a large labeled corpus. However,
little guidance exists on deploying these methods to improve production ASR
systems that are trained on very large supervised corpora and with realistic
requirements like a constrained model size and CPU budget, streaming
capability, and a rich lattice for rescoring and for downstream NLU tasks. In
this work, we compare three state-of-the-art semi-supervised methods
encompassing both unpaired text and audio as well as several of their
combinations in a controlled setting using joint training. We find that in our
setting these methods offer many improvements beyond raw WER, including
substantial gains in tail-word WER, decoder computation during inference, and
lattice density.

比较三种基于半监督方法的语音识别性能提高策略

大规模流式 ASR 半监督学习技术比较

A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at  Scale

We propose JEIT, a joint end-to-end (E2E) model and internal language model
(ILM) training method to inject large-scale unpaired text into ILM during E2E
training which improves rare-word speech recognition. With JEIT, the E2E model
computes an E2E loss on audio-transcript pairs while its ILM estimates a
cross-entropy loss on unpaired text. The E2E model is trained to minimize a
weighted sum of E2E and ILM losses. During JEIT, ILM absorbs knowledge from
unpaired text while the E2E training serves as regularization. Unlike ILM
adaptation methods, JEIT does not require a separate adaptation step and avoids
the need for Kullback-Leibler divergence regularization of ILM. We also show
that modular hybrid autoregressive transducer (MHAT) performs better than HAT
in the JEIT framework, and is much more robust than HAT during ILM adaptation.
To push the limit of unpaired text injection, we further propose a combined
JEIT and JOIST training (CJJT) that benefits from modality matching, encoder
text injection and ILM training. Both JEIT and CJJT can foster a more effective
LM fusion. With 100B unpaired sentences, JEIT/CJJT improves rare-word
recognition accuracy by up to 16.4% over a model trained without unpaired text.

提出了一种使用大规模非配对文本来改善说话者对罕见单词的识别能力的训练方法 JEIT，它结合了端到端模型和内部语言模型训练，其中内部语言模型吸收非配对文本，接受 E2E 计算损失信息，从而提高 E2E 模型的性能。在 JEIT 过程中，MHAT 表现比 HAT 更佳，在 ILM 适应时更加稳定，此外，我们还提出了结合 JEIT 和 JOIST 的 CJJT 方法，它能更有效地实现语言模型的融合。