Traditionally, NLP performance improvement has been focused on improving models and increasing the number of model parameters. NLP vocabulary construction has remained focused on maximizing the number of words represented through subword regularization. We present a novel tokenizer that uses semantics to drive vocabulary construction. The tokenizer includes a trainer that uses stemming to enhance subword formation. Further optimizations and adaptations are implemented to minimize the number of words that cannot be encoded. The encoder is updated to integrate with the trainer. The tokenizer is implemented as a drop-in replacement for the SentencePiece tokenizer. The new tokenizer more than doubles the number of wordforms represented in the vocabulary. The enhanced vocabulary significantly improves NLP model convergence, and improves quality of word and sentence embeddings. Our experimental results show top performance on two Glue tasks using BERT-base, improving on models more than 50X in size.

本研究提出了一种基于语义的新型分词器，使用词干来增强子词的形成，最小化未编码的单词数量，对接Trainer，实现了比SentencePiece分词器更高效的分词，并在BERT模型上实验，表明此方法可以将单词数量提高一倍以上，并显着提高NLP模型的收敛速度和单词及句子嵌入的质量，成绩排名Glue任务之前，比同等规模的模型优越50倍以上。

增强自然语言处理的语义分词器