We present a suite of interventions and experiments that allow us to
understand language model adaptation to text with linguistic variation (e.g.,
nonstandard or dialectal text). Our interventions address several features of
linguistic variation, resulting in character, subword, and word-level changes.
Applying our interventions during language model adaptation with varying size
and nature of training data, we gain important insights into what makes
linguistic variation particularly difficult for language models to deal with.
For instance, on text with character-level variation, performance improves with
even a few training examples but approaches a plateau, suggesting that more
data is not the solution. In contrast, on text with variation involving new
words or meanings, far more data is needed, but it leads to a massive
breakthrough in performance. Our findings inform future work on dialectal NLP
and making language models more robust to linguistic variation overall. We make
the code for our interventions, which can be applied to any English text data,
publicly available.

通过一系列干预和实验证明，我们可以理解语言模型对于存在语言变异（例如非标准或方言文本）的文本的适应性。在包括字符级、子词级和词级变化的语言变异方面进行干预，通过不同规模和性质的训练数据进行语言模型适应，我们对于语言变异对于语言模型的困难有了重要的认识。我们的发现对于方言自然语言处理和增强语言模型对语言变异的鲁棒性的未来研究有着重要的启示。我们公开提供了可以应用于任何英文文本数据的干预代码。

我们正在呼唤干预：对语言模型在不同类型语言变异中的适应性进行深入研究

We're Calling an Intervention: Taking a Closer Look at Language Model  Adaptation to Different Types of Linguistic Variation

Despite recent advances, Automatic Speech Recognition (ASR) systems are still
far from perfect. Typical errors include acronyms, named entities and
domain-specific special words for which little or no data is available. To
address the problem of recognizing these words, we propose an self-supervised
continual learning approach. Given the audio of a lecture talk with
corresponding slides, we bias the model towards decoding new words from the
slides by using a memory-enhanced ASR model from previous work. Then, we
perform inference on the talk, collecting utterances that contain detected new
words into an adaptation dataset. Continual learning is then performed on this
set by adapting low-rank matrix weights added to each weight matrix of the
model. The whole procedure is iterated for many talks. We show that with this
approach, we obtain increasing performance on the new words when they occur
more frequently (more than 80% recall) while preserving the general performance
of the model.

基于自我监督的持续学习方法，在自动语音识别系统中识别新词时表现出增长性能，通过使用来自之前工作中的增强型记忆自动语音识别模型，通过偏置模型向滑动中的新词进行解码，并对检测到的新词使用推理，将包含这些新词的话语收集到自适应数据集中，对该集进行持续学习，通过调整加到每个模型的权重矩阵上的低秩矩阵权重。该方法在新词经常出现时获得递增性能（超过 80% 的召回率），同时保持模型的一般性能。