Recent language models have shown impressive multilingual performance, even
when not explicitly trained for it. Despite this, concerns have been raised
about the quality of their outputs across different languages. In this paper,
we show how disparity in the treatment of different languages arises at the
tokenization stage, well before a model is even invoked. The same text
translated into different languages can have drastically different tokenization
lengths, with differences up to 15 times in some cases. These disparities
persist across the 17 tokenizers we evaluate, even if they are intentionally
trained for multilingual support. Character-level and byte-level models also
exhibit over 4 times the difference in the encoding length for some language
pairs. This induces unfair treatment for some language communities in regard to
the cost of accessing commercial language services, the processing time and
latency, as well as the amount of content that can be provided as context to
the models. Therefore, we make the case that we should train future language
models using multilingually fair tokenizers.

研究显示，语言模型在处理不同语言时存在标记化差异，导致部分语言社区无法公平访问商业语言服务、处理时间和网络延迟，并限制了模型学习的上下文，因此建议使用多语种公平标记器来训练未来的语言模型。