Despite recent advances, Automatic Speech Recognition (ASR) systems are still
far from perfect. Typical errors include acronyms, named entities and
domain-specific special words for which little or no data is available. To
address the problem of recognizing these words, we propose an self-supervised
continual learning approach. Given the audio of a lecture talk with
corresponding slides, we bias the model towards decoding new words from the
slides by using a memory-enhanced ASR model from previous work. Then, we
perform inference on the talk, collecting utterances that contain detected new
words into an adaptation dataset. Continual learning is then performed on this
set by adapting low-rank matrix weights added to each weight matrix of the
model. The whole procedure is iterated for many talks. We show that with this
approach, we obtain increasing performance on the new words when they occur
more frequently (more than 80% recall) while preserving the general performance
of the model.

基于自我监督的持续学习方法，在自动语音识别系统中识别新词时表现出增长性能，通过使用来自之前工作中的增强型记忆自动语音识别模型，通过偏置模型向滑动中的新词进行解码，并对检测到的新词使用推理，将包含这些新词的话语收集到自适应数据集中，对该集进行持续学习，通过调整加到每个模型的权重矩阵上的低秩矩阵权重。该方法在新词经常出现时获得递增性能（超过 80% 的召回率），同时保持模型的一般性能。