Despite it being the cornerstone of BPE, the most common tokenization
algorithm, the importance of compression in the tokenization process is still
unclear. In this paper, we argue for the theoretical importance of compression,
that can be viewed as 0-gram language modeling where equal probability is
assigned to all tokens. We also demonstrate the empirical importance of
compression for downstream success of pre-trained language models. We control
the compression ability of several BPE tokenizers by varying the amount of
documents available during their training: from 1 million documents to a
character-based tokenizer equivalent to no training data at all. We then
pre-train English language models based on those tokenizers and fine-tune them
over several tasks. We show that there is a correlation between tokenizers'
compression and models' downstream performance, suggesting that compression is
a reliable intrinsic indicator of tokenization quality. These correlations are
more pronounced for generation tasks (over classification) or for smaller
models (over large ones). We replicated a representative part of our
experiments on Turkish and found similar results, confirming that our results
hold for languages with typological characteristics dissimilar to English. We
conclude that building better compressing tokenizers is a fruitful avenue for
further research and for improving overall model performance.

通过变化训练数据的数量，我们研究了 BPE tokenizers 的压缩能力对预训练语言模型下游性能的影响，我们发现压缩能力与模型性能存在相关性，因此构建压缩效果更好的 tokenizer 是一个有前景的研究方向。