Morphological inflection is a process of word formation where base words are modified to express different grammatical categories such as tense, case, voice, person, or number. World Englishes, such as Colloquial Singapore English (CSE) and African American Vernacular English (AAVE), differ from Standard English dialects in inflection use. Although comprehension by human readers is usually unimpaired by non-standard inflection use, NLP systems are not so robust. We introduce a new Base-Inflection Encoding of English text that is achieved by combining linguistic and statistical techniques. Fine-tuning pre-trained NLP models for downstream tasks under this novel encoding achieves robustness to non-standard inflection use while maintaining performance on Standard English examples. Models using this encoding also generalize better to non-standard dialects without explicit training. We suggest metrics to evaluate tokenizers and extensive model-independent analyses demonstrate the efficacy of the encoding when used together with data-driven subword tokenizers.

该论文提出了基于BITE的英文文本分词方法，将屈折词语还原为其基本形式，并通过添加特殊符号重新注入语法信息，提高了NLP模型在非标准屈折形式方面的鲁棒性并保持了在干净数据上的性能。研究者们通过评估提出了度量语料库效率的指标。

注意词形变化！基于词形变化编码改进非标准英语的自然语言处理