Despite the significant progress in end-to-end (E2E) automatic speech
recognition (ASR), E2E ASR for low resourced code-switching (CS) speech has not
been well studied. In this work, we describe an E2E ASR pipeline for the
recognition of CS speech in which a low-resourced language is mixed with a high
resourced language. Low-resourcedness in acoustic data hinders the performance
of E2E ASR systems more severely than the conventional ASR systems.~To mitigate
this problem in the transcription of archives with code-switching Frisian-Dutch
speech, we integrate a designated decoding scheme and perform rescoring with
neural network-based language models to enable better utilization of the
available textual resources. We first incorporate a multi-graph decoding
approach which creates parallel search spaces for each monolingual and mixed
recognition tasks to maximize the utilization of the textual resources from
each language. Further, language model rescoring is performed using a recurrent
neural network pre-trained with cross-lingual embedding and further adapted
with the limited amount of in-domain CS text. The ASR experiments demonstrate
the effectiveness of the described techniques in improving the recognition
performance of an E2E CS ASR system in a low-resourced scenario.

本文研究了一种面向代码交错语音的端到端语音识别技术，并利用神经网络语言模型和循环神经网络提高低资源情况下的识别性能。

低资源语言对的端到端代码开关 ASR

End-to-End Code-Switching ASR for Low-Resourced Language Pairs

Direct acoustics-to-word (A2W) systems for end-to-end automatic speech
recognition are simpler to train, and more efficient to decode with, than
sub-word systems. However, A2W systems can have difficulties at training time
when data is limited, and at decoding time when recognizing words outside the
training vocabulary. To address these shortcomings, we investigate the use of
recently proposed acoustic and acoustically grounded word embedding techniques
in A2W systems. The idea is based on treating the final pre-softmax weight
matrix of an AWE recognizer as a matrix of word embedding vectors, and using an
externally trained set of word embeddings to improve the quality of this
matrix. In particular we introduce two ideas: (1) Enforcing similarity at
training time between the external embeddings and the recognizer weights, and
(2) using the word embeddings at test time for predicting out-of-vocabulary
words. Our word embedding model is acoustically grounded, that is it is learned
jointly with acoustic embeddings so as to encode the words' acoustic-phonetic
content; and it is parametric, so that it can embed any arbitrary (potentially
out-of-vocabulary) sequence of characters. We find that both techniques improve
the performance of an A2W recognizer on conversational telephone speech.

通过将最终的 AWE 识别器的 pre-softmax 权重矩阵视为词嵌入向量矩阵，并利用外部训练的词嵌入向量来改善其质量，我们提出了两种思路：（1）在训练中在外部嵌入向量与识别器权重之间强制相似性，（2）在测试时使用词嵌入向量预测生僻词以提高直接声学到单词的性能。