Due to the dynamic nature of human language, automatic speech recognition
(ASR) systems need to continuously acquire new vocabulary. Out-Of-Vocabulary
(OOV) words, such as trending words and new named entities, pose problems to
modern ASR systems that require long training times to adapt their large
numbers of parameters. Different from most previous research focusing on
language model post-processing, we tackle this problem on an earlier processing
level and eliminate the bias in acoustic modeling to recognize OOV words
acoustically. We propose to generate OOV words using text-to-speech systems and
to rescale losses to encourage neural networks to pay more attention to OOV
words. Specifically, we enlarge the classification loss used for training
neural networks' parameters of utterances containing OOV words
(sentence-level), or rescale the gradient used for back-propagation for OOV
words (word-level), when fine-tuning a previously trained model on synthetic
audio. To overcome catastrophic forgetting, we also explore the combination of
loss rescaling and model regularization, i.e. L2 regularization and elastic
weight consolidation (EWC). Compared with previous methods that just fine-tune
synthetic audio with EWC, the experimental results on the LibriSpeech benchmark
reveal that our proposed loss rescaling approach can achieve significant
improvement on the recall rate with only a slight decrease on word error rate.
Moreover, word-level rescaling is more stable than utterance-level rescaling
and leads to higher recall rates and precision on OOV word recognition.
Furthermore, our proposed combined loss rescaling and weight consolidation
methods can support continual learning of an ASR system.

本篇论文提出了一种利用文本到语音系统生成 OOV 单词并通过损失调整和正则化实现语音识别系统的持续学习，并且相比于之前的方法，该方法可以实现更高的召回率和更高的准确性。