The end-to-end ASR model is often desired in the streaming multilingual
scenario since it is easier to deploy and can benefit from pre-trained speech
models such as powerful foundation models. Meanwhile, the heterogeneous nature
and imbalanced data abundance of different languages may cause performance
degradation, leading to asynchronous peak performance for different languages
during training, especially on tail ones. Sometimes even the data itself may
become unavailable as a result of the enhanced privacy protection. Existing
work tend to significantly increase the model size or learn language-specific
decoders to accommodate each language separately. In this study, we explore
simple yet effective Language-Dependent Adapter (LDA) finetuning under a
cascaded Conformer transducer framework enhanced by teacher pseudo-labeling for
tail languages in the streaming multilingual ASR. The adapter only accounts for
0.4% of the full model per language. It is plugged into the frozen foundation
model and is the only trainable module during the finetuning process with noisy
student training. The final model merges the adapter parameters from different
checkpoints for different languages. The model performance is validated on a
challenging multilingual dictation dataset, which includes 39 tail languages
across Latin, Greek, Arabic, etc. Our proposed method brings 12.2% word error
rate reduction on average and up to 37.5% on a single locale. Furthermore, we
show that our parameter-efficient LDA can match the quality of the full model
finetuning, thus greatly alleviating the asynchronous peak performance issue.

通过在流式多语种 ASR 中引入语言相关适配器（LDA）微调和基于教师伪标签技术，本研究提出的方法有效地减少了平均 12.2% 的词错误率，并在单个场景中最多降低了 37.5% 的错误率，从而显著缓解了异步峰值性能问题。

流式多语种 ASR 中对尾语言的高效适配器微调

Efficient Adapter Finetuning for Tail Languages in Streaming  Multilingual ASR

To let the state-of-the-art end-to-end ASR model enjoy data efficiency, as
well as much more unpaired text data by multi-modal training, one needs to
address two problems: 1) the synchronicity of feature sampling rates between
speech and language (aka text data); 2) the homogeneity of the learned
representations from two encoders. In this paper we propose to employ a novel
bidirectional attention mechanism (BiAM) to jointly learn both ASR encoder
(bottom layers) and text encoder with a multi-modal learning method. The BiAM
is to facilitate feature sampling rate exchange, realizing the quality of the
transformed features for the one kind to be measured in another space, with
diversified objective functions. As a result, the speech representations are
enriched with more linguistic information, while the representations generated
by the text encoder are more similar to corresponding speech ones, and
therefore the shared ASR models are more amenable for unpaired text data
pretraining. To validate the efficacy of the proposed method, we perform two
categories of experiments with or without extra unpaired text data.
Experimental results on Librispeech corpus show it can achieve up to 6.15% word
error rate reduction (WERR) with only paired data learning, while 9.23% WERR
when more unpaired text data is employed.

采用双向注意力机制的多模态学习方法，能够有效地提升语音表示的语言信息，增强文本的语音表示，从而使共享的 ASR 模型更适用于无配对的文本数据预训练，仅使用配对数据学习时，单词错误率减少了 6.15％，使用更多无配对文本数据时，错误率减少了 9.23％。

基于双向注意力的语音文本多模态训练，提高语音识别能力

Speech-text based multi-modal training with bidirectional attention for improved speech recognition

An end-to-end (E2E) ASR model implicitly learns a prior Internal Language
Model (ILM) from the training transcripts. To fuse an external LM using Bayes
posterior theory, the log likelihood produced by the ILM has to be accurately
estimated and subtracted. In this paper we propose two novel approaches to
estimate the ILM based on Listen-Attend-Spell (LAS) framework. The first method
is to replace the context vector of the LAS decoder at every time step with a
vector that is learned with training transcripts. Furthermore, we propose
another method that uses a lightweight feed-forward network to directly map
query vector to context vector in a dynamic sense. Since the context vectors
are learned by minimizing the perplexities on training transcripts, and their
estimation is independent of encoder output, hence the ILMs are accurately
learned for both methods. Experiments show that the ILMs achieve the lowest
perplexity, indicating the efficacy of the proposed methods. In addition, they
also significantly outperform the shallow fusion method, as well as two
previously proposed ILM Estimation (ILME) approaches on several datasets.

本文提出了两种新方法，一种是基于训练文本学习的上下文向量替换方法，另一种是基于轻量级前馈网络的动态映射方法来准确估计内部语言模型 (ILM)，实验结果显示这两种方法比传统浅层融合和其他 ILM 估计方法表现更好。