Tokenization is an understudied and often neglected component of modern LLMs.
Most published works use a single tokenizer for all experiments, often borrowed
from another model, without performing ablations or analysis to optimize
tokenization. Moreover, the tokenizer is generally kept unchanged when
fine-tuning a base model. In this paper, we show that the size,
pre-tokenization regular expression, and training data of a tokenizer can
significantly impact the model's generation speed, effective context size,
memory usage, and downstream performance. We train specialized Byte-Pair
Encoding code tokenizers, and conduct extensive ablations on the impact of
tokenizer design on the performance of LLMs for code generation tasks such as
HumanEval and MBPP, and provide recommendations for tokenizer hyper-parameters
selection and switching the tokenizer in a pre-trained LLM. We perform our
experiments on models trained from scratch and from pre-trained models,
verifying their applicability to a wide range of use-cases. We find that when
fine-tuning on more than 50 billion tokens, we can specialize the tokenizer of
a pre-trained LLM to obtain large gains in generation speed and effective
context size.

我们的研究表明，分词对于现代 LLMs 至关重要，而且往往被忽视。我们通过在代码生成任务中对 tokenizer 设计进行了详细实验，并提出了分词器超参数选择和在预训练 LLMs 中切换分词器的建议。我们发现，在超过 500 亿个标记的细调过程中，可以对预训练 LLMs 的分词器进行专门优化，从而显著提高生成速度和有效上下文大小。