The popular subword tokenizers of current language models, such as Byte-Pair Encoding (BPE), are known not to respect morpheme boundaries, which affects the downstream performance of the models. While many improved tokenization algorithms have been proposed, their evaluation and cross-comparison is still an open problem. As a solution, we propose a combined intrinsic-extrinsic evaluation framework for subword tokenization. Intrinsic evaluation is based on our new UniMorph Labeller tool that classifies subword tokenization as either morphological or alien. Extrinsic evaluation, in turn, is performed via the Out-of-Vocabulary Generalization Challenge 1.0 benchmark, which consists of three newly specified downstream text classification tasks. Our empirical findings show that the accuracy of UniMorph Labeller is 98%, and that, in all language models studied (including ALBERT, BERT, RoBERTa, and DeBERTa), alien tokenization leads to poorer generalizations compared to morphological tokenization for semantic compositionality of word meanings.

我们提出了一种联合内在-外在评估框架用于子词切分，通过UniMorph Labeller工具对子词切分进行内在评估，将其分类为形态还是外语；然后通过Out-of-Vocabulary Generalization Challenge 1.0基准进行外在评估，包括三个新指定的下游文本分类任务；我们的实证研究发现，UniMorph Labeller的准确率达到了98%；而在所有语言模型（包括ALBERT、BERT、RoBERTa和DeBERTa）中，与形态切分相比，外语切分导致了更差的泛化能力，影响了词义的语义组合性。

评估次词标记化：外语次词组合与OOV泛化挑战