End-to-end models, such as the neural Transducer, have been successful in
integrating acoustic and linguistic information jointly to achieve excellent
recognition performance. However, adapting these models with text-only data is
challenging. Factorized neural Transducer (FNT) aims to address this issue by
introducing a separate vocabulary decoder to predict the vocabulary, which can
effectively perform traditional text data adaptation. Nonetheless, this
approach has limitations in fusing acoustic and language information
seamlessly. Moreover, a degradation in word error rate (WER) on the general
test sets was also observed, leading to doubts about its overall performance.
In response to this challenge, we present an improved factorized neural
Transducer (IFNT) model structure designed to comprehensively integrate
acoustic and language information while enabling effective text adaptation. We
evaluate the performance of our proposed methods through in-domain experiments
on GigaSpeech and out-of-domain experiments adapting to EuroParl, TED-LIUM, and
Medical datasets. After text-only adaptation, IFNT yields 7.9% to 28.5%
relative WER improvements over the standard neural Transducer with shallow
fusion, and relative WER reductions ranging from 1.6% to 8.2% on the three test
sets compared to the FNT model.

通过设立独立语料库解码器预测词汇，改进因式神经变换器（IFNT）模型结构综合整合声学和语言信息，实现有效的文本适应，并在实验中展现了相对于标准神经变换器与浅层融合模型的 7.9% 至 28.5% 的相对词误率改善和相对于 FNT 模型的 1.6% 至 8.2% 的词误率降低。

文本领域适应的改进因式化神经转导模型

Improved Factorized Neural Transducer Model For text-only Domain  Adaptation

In spite of the excellent strides made by end-to-end (E2E) models in speech
recognition in recent years, named entity recognition is still challenging but
critical for semantic understanding. In order to enhance the ability to
recognize named entities in E2E models, previous studies mainly focus on
various rule-based or attention-based contextual biasing algorithms. However,
their performance might be sensitive to the biasing weight or degraded by
excessive attention to the named entity list, along with a risk of false
triggering. Inspired by the success of the class-based language model (LM) in
named entity recognition in conventional hybrid systems and the effective
decoupling of acoustic and linguistic information in the factorized neural
Transducer (FNT), we propose a novel E2E model to incorporate class-based LMs
into FNT, which is referred as C-FNT. In C-FNT, the language model score of
named entities can be associated with the name class instead of its surface
form. The experimental results show that our proposed C-FNT presents
significant error reduction in named entities without hurting performance in
general word recognition.

为了提高端到端模型中的实体识别能力，本研究将基于类别的语言模型应用于因式分解的神经递归式转录器（FNT）中，取得显著的实体识别误差降低。

在分解神经传输器中融入基于类别的语言模型的命名实体识别

Incorporating Class-based Language Model for Named Entity Recognition in  Factorized Neural Transducer

Neural transducer is now the most popular end-to-end model for speech
recognition, due to its naturally streaming ability. However, it is challenging
to adapt it with text-only data. Factorized neural transducer (FNT) model was
proposed to mitigate this problem. The improved adaptation ability of FNT on
text-only adaptation data came at the cost of lowered accuracy compared to the
standard neural transducer model. We propose several methods to improve the
performance of the FNT model. They are: adding CTC criterion during training,
adding KL divergence loss during adaptation, using a pre-trained language model
to seed the vocabulary predictor, and an efficient adaptation approach by
interpolating the vocabulary predictor with the n-gram language model. A
combination of these approaches results in a relative word-error-rate reduction
of 9.48\% from the standard FNT model. Furthermore, n-gram interpolation with
the vocabulary predictor improves the adaptation speed hugely with satisfactory
adaptation performance.

介绍了因文本数据处理能力有限而提出的 FNT 模型及其在语音识别领域中的存在意义，并提出了多种方法来优化模型性能，使用这些方法后，相比于标准 FNT 模型，模型的错误率下降了 9.48%。