The recent success of LLMs has been predominantly driven by curating the
training dataset composition, scaling of model architectures and dataset sizes
and advancements in pretraining objectives, leaving tokenizer influence as a
blind spot. Shedding light on this underexplored area, we conduct a
comprehensive study on the influence of tokenizer choice on LLM downstream
performance by training 24 mono- and multilingual LLMs at a 2.6B parameter
scale, ablating different tokenizer algorithms and parameterizations. Our
studies highlight that the tokenizer choice can significantly impact the
model's downstream performance, training and inference costs. In particular, we
find that the common tokenizer evaluation metrics fertility and parity are not
always predictive of model downstream performance, rendering these metrics a
questionable choice for tokenizer evaluation. Furthermore, we show that
multilingual tokenizers trained on the five most frequent European languages
require vocabulary size increases of factor three in comparison to English.
While English-only tokenizers have been applied to the training of
multi-lingual LLMs in the past, we find that this approach results in a severe
downstream performance degradation and additional training costs of up to 68%,
due to an inefficient tokenization vocabulary.

在大规模预训练语言模型（LLMs）中，我们进行了一项全面研究，探究了分词器选择对下游性能、训练数据集和词汇量的影响。研究发现，分词器的选择可以显著影响模型的下游性能、训练和推理成本，并指出常用的分词器评估指标并不能准确预测模型性能，特别是多语言分词器在词汇量方面需要比英语增加三倍。在训练多语言 LLMs 时，仅采用英语分词器会导致严重的性能下降和高达 68% 的额外训练成本，因为其分词词汇表效率低下。