The use of low-rank adaptation (LoRA) with frozen pretrained language models
(PLMs) has become increasing popular as a mainstream, resource-efficient
modeling approach for memory-constrained hardware. In this study, we first
explore how to enhance model performance by introducing various LoRA training
strategies, achieving relative word error rate reductions of 3.50\% on the
public Librispeech dataset and of 3.67\% on an internal dataset in the
messaging domain. To further characterize the stability of LoRA-based
second-pass speech recognition models, we examine robustness against input
perturbations. These perturbations are rooted in homophone replacements and a
novel metric called N-best Perturbation-based Rescoring Robustness (NPRR), both
designed to measure the relative degradation in the performance of rescoring
models. Our experimental results indicate that while advanced variants of LoRA,
such as dynamic rank-allocated LoRA, lead to performance degradation in
$1$-best perturbation, they alleviate the degradation in $N$-best perturbation.
This finding is in comparison to fully-tuned models and vanilla LoRA tuning
baselines, suggesting that a comprehensive selection is needed when using
LoRA-based adaptation for compute-cost savings and robust language modeling.

使用低秩适应（LoRA）和预训练语言模型（PLMs）的方法已成为一种流行的资源高效性建模方法，本研究首先探讨了通过引入各种 LoRA 培训策略来提高模型性能，相对词错误率减少了 3.50％和 3.67％，并且进一步检验了 LoRA 基于二次通过语音识别模型的稳定性和对输入扰动的鲁棒性。

语音识别中语言建模的低秩适应的训练策略和模型鲁棒性研究探究

Investigating Training Strategies and Model Robustness of Low-Rank  Adaptation for Language Modeling in Speech Recognition

In streaming settings, speech recognition models have to map sub-sequences of
speech to text before the full audio stream becomes available. However, since
alignment information between speech and text is rarely available during
training, models need to learn it in a completely self-supervised way. In
practice, the exponential number of possible alignments makes this extremely
challenging, with models often learning peaky or sub-optimal alignments. Prima
facie, the exponential nature of the alignment space makes it difficult to even
quantify the uncertainty of a model's alignment distribution. Fortunately, it
has been known for decades that the entropy of a probabilistic finite state
transducer can be computed in time linear to the size of the transducer via a
dynamic programming reduction based on semirings. In this work, we revisit the
entropy semiring for neural speech recognition models, and show how alignment
entropy can be used to supervise models through regularization or distillation.
We also contribute an open-source implementation of CTC and RNN-T in the
semiring framework that includes numerically stable and highly parallel
variants of the entropy semiring. Empirically, we observe that the addition of
alignment distillation improves the accuracy and latency of an already
well-optimized teacher-student distillation model, achieving state-of-the-art
performance on the Librispeech dataset in the streaming scenario.

在流媒体环境中，语音识别模型需要在完整音频流可用之前将语音的子序列映射为文本，但由于训练过程中很少有语音和文本之间的对齐信息可用，模型需要以完全自我监督的方式进行学习。本文通过熵半环重新考虑了神经语音识别模型，展示了如何通过对齐熵来通过正则化或蒸馏来监督模型。在实证上，我们观察到对齐蒸馏的添加可以提高已经经过优化的师生蒸馏模型的准确性和延迟，实现了在流媒体场景下 Librispeech 数据集上的最新性能。

重新考虑神经语音识别中的熵半环

Revisiting the Entropy Semiring for Neural Speech Recognition

This paper introduces a new dataset, Libri-Adapt, to support unsupervised
domain adaptation research on speech recognition models. Built on top of the
LibriSpeech corpus, Libri-Adapt contains English speech recorded on mobile and
embedded-scale microphones, and spans 72 different domains that are
representative of the challenging practical scenarios encountered by ASR
models. More specifically, Libri-Adapt facilitates the study of domain shifts
in ASR models caused by a) different acoustic environments, b) variations in
speaker accents, c) heterogeneity in the hardware and platform software of the
microphones, and d) a combination of the aforementioned three shifts. We also
provide a number of baseline results quantifying the impact of these domain
shifts on the Mozilla DeepSpeech2 ASR model.

本文介绍了一个新的数据集 Libri-Adapt，它是 LibriSpeech 语料库的扩展，包含来自 72 个不同领域的英语语音，可以支持语音识别模型的无监督领域自适应研究，并提供基线结果来量化这些领域转移对 Mozilla DeepSpeech2 ASR 模型的影响。