Language models typically tokenize raw text into sequences of subword
identifiers from a predefined vocabulary, a process inherently sensitive to
typographical errors, length variations, and largely oblivious to the internal
structure of tokens-issues we term the curse of tokenization. In this study, we
delve into these drawbacks and demonstrate that large language models (LLMs)
remain susceptible to these problems. This study systematically investigates
these challenges and their impact on LLMs through three critical research
questions: (1) complex problem solving, (2) token structure probing, and (3)
resilience to typographical variation. Our findings reveal that scaling model
parameters can mitigate the issue of tokenization; however, LLMs still suffer
from biases induced by typos and other text format variations. Our experiments
show that subword regularization such as BPE-dropout can mitigate this issue.
We will release our code and data to facilitate further research.

大型语言模型存在分词问题，导致对错别字、长度差异和标记内部结构的忽视。本研究通过探究复杂问题解决、标记结构探测和对错别字的抵抗力来系统地调查这些挑战及其对大型语言模型的影响，并展示模型参数缩放与子词规范化对解决这些问题的作用。

分词不足：分词的诅咒

Tokenization Falling Short: The Curse of Tokenization

Typographical errors are a major source of frustration for visitors of online
marketplaces. Because of the domain-specific nature of these marketplaces and
the very short queries users tend to search for, traditional spell cheking
solutions do not perform well in correcting typos. We present a data
augmentation method to address the lack of annotated typo data and train a
recurrent neural network to learn context-limited domain-specific embeddings.
Those embeddings are deployed in a real-time inferencing API for the Microsoft
AppSource marketplace to find the closest match between a misspelled user query
and the available product names. Our data efficient solution shows that
controlled high quality synthetic data may be a powerful tool especially
considering the current climate of large language models which rely on
prohibitively huge and often uncontrolled datasets.

通过数据增强和递归神经网络，我们提出了一种解决在线市场拼写错误的方法，以找到用户查询和可用产品名称之间的最佳匹配。这种数据高效的解决方案证明了受控高质量合成数据在当前依赖于庞大且常常无法控制的数据集的大型语言模型的背景下可能成为强大的工具。