Multilingual language models have recently gained attention as a promising
solution for representing multiple languages in a single model. In this paper,
we propose new criteria to evaluate the quality of lexical representation and
vocabulary overlap observed in sub-word tokenizers. Our findings show that the
overlap of vocabulary across languages can be actually detrimental to certain
downstream tasks (POS, dependency tree labeling). In contrast, NER and
sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing
vocabulary. We also observe that the coverage of the language-specific tokens
in the multilingual vocabulary significantly impacts the word-level tasks. Our
study offers a deeper understanding of the role of tokenizers in multilingual
language models and guidelines for future model developers to choose the most
suitable tokenizer for their specific application before undertaking costly
model pre-training

本文提出新的标准以评估子词符号化器中的词汇表示质量和词汇重叠度，并发现跨语言单词表的重叠实际上可能对某些下游任务（如 POS、依赖树标记）产生负面影响，而在命名实体识别和句子级任务（如跨语言检索、NLI）中分享词汇表是有益的。此外，本文还观察到多语种语言模型中特定语言标记的覆盖范围显著影响单词级任务。因此，我们为未来的模型开发人员提供了详细的指导，以选择最适合他们特定应用程序的符号化器。