A multilingual tokenizer is a fundamental component of multilingual neural machine translation. It is trained from a multilingual corpus. Since a skewed data distribution is considered to be harmful, a sampling strategy is usually used to balance languages in the corpus. However, few works have systematically answered how language imbalance in tokenizer training affects downstream performance. In this work, we analyze how translation performance changes as the data ratios among languages vary in the tokenizer training corpus. We find that while relatively better performance is often observed when languages are more equally sampled, the downstream performance is more robust to language imbalance than we usually expected. Two features, UNK rate and closeness to the character level, can warn of poor downstream performance before performing the task. We also distinguish language sampling for tokenizer training from sampling for model training and show that the model is more sensitive to the latter.

本研究分析研究了多语言经过语言不平衡的训练语料库的分词器，并发现在训练过程中，UNK率和接近字符级别的特征对于下游任务的性能具有预警作用。同时，该研究还将分词器训练的语言采样与模型训练的采样区分开，并指出模型对后者更为敏感。

神经机器翻译对多语言分词训练中的语言不平衡的鲁棒性有多强？