Modern large language models demonstrate impressive capabilities in text
generation and generalization. However, they often struggle with solving text
editing tasks, particularly when it comes to correcting spelling errors and
mistypings. In this paper, we present a methodology for generative spelling
correction (SC), which was tested on English and Russian languages and
potentially can be extended to any language with minor changes. Our research
mainly focuses on exploring natural spelling errors and mistypings in texts and
studying the ways those errors can be emulated in correct sentences to
effectively enrich generative models' pre-train procedure. We investigate the
impact of such emulations and the models' abilities across different text
domains. In this work, we investigate two spelling corruption techniques: 1)
first one mimics human behavior when making a mistake through leveraging
statistics of errors from particular dataset and 2) second adds the most common
spelling errors, keyboard miss clicks, and some heuristics within the texts. We
conducted experiments employing various corruption strategies, models'
architectures and sizes on the pre-training and fine-tuning stages and
evaluated the models using single-domain and multi-domain test sets. As a
practical outcome of our work, we introduce SAGE (Spell checking via
Augmentation and Generative distribution Emulation) is a library for automatic
generative SC that includes a family of pre-trained generative models and
built-in augmentation algorithms.

我们提出了一种对生成模型进行拼写纠错的方法，该方法通过研究文本中的自然拼写错误和打字错误，探索了模拟这些错误在正确句子中的方式，以有效丰富生成模型的预训练过程，并研究了这些模拟对模型能力的影响与文本域的关系。