Tokenization significantly influences language models(LMs)' performance. This
paper traces the evolution of tokenizers from word-level to subword-level,
analyzing how they balance tokens and types to enhance model adaptability while
controlling complexity. Despite subword tokenizers like Byte Pair Encoding
(BPE) overcoming many word tokenizer limitations, they encounter difficulties
in handling non-Latin languages and depend heavily on extensive training data
and computational resources to grasp the nuances of multiword expressions
(MWEs). This article argues that tokenizers, more than mere technical tools,
should drawing inspiration from the cognitive science about human language
processing. This study then introduces the "Principle of Least Effort" from
cognitive science, that humans naturally seek to reduce cognitive effort, and
discusses the benefits of this principle for tokenizer development. Based on
this principle, the paper proposes that the Less-is-Better (LiB) model could be
a new approach for LLM tokenizer. The LiB model can autonomously learn an
integrated vocabulary consisting of subwords, words, and MWEs, which
effectively reduces both the numbers of tokens and types. Comparative
evaluations show that the LiB tokenizer outperforms existing word and BPE
tokenizers, presenting an innovative method for tokenizer development, and
hinting at the possibility of future cognitive science-based tokenizers being
more efficient.

通过追踪分词器从词级到子词级的演变，本研究分析了分词器如何在增强模型适应性的同时控制复杂度方面平衡标记和类型。根据认知科学中的 “最省力原则”，本文提出了一种新的 LLN 分词器 LiB 模型，它可以自主学习由子词、词和多词表达式组成的综合词汇，从而有效减少标记和类型的数量。比较评估显示，LiB 分词器优于现有的词级和 BPE 分词器，为分词器开发提供了创新方法，并提示未来的基于认知科学的分词器可能更为高效。

重新思考分词：为大型语言模型打造更好的分词器

Rethinking Tokenization: Crafting Better Tokenizers for Large Language  Models

Language identification for code-switching (CS), the phenomenon of
alternating between two or more languages in conversations, has traditionally
been approached under the assumption of a single language per token. However,
if at least one language is morphologically rich, a large number of words can
be composed of morphemes from more than one language (intra-word CS). In this
paper, we extend the language identification task to the subword-level, such
that it includes splitting mixed words while tagging each part with a language
ID. We further propose a model for this task, which is based on a segmental
recurrent neural network. In experiments on a new Spanish--Wixarika dataset and
on an adapted German--Turkish dataset, our proposed model performs slightly
better than or roughly on par with our best baseline, respectively. Considering
only mixed words, however, it strongly outperforms all baselines.

本文将语种识别任务推广到子词级别，研究如何识别包含两种以上语言的语素（intra-word CS），提出了一种基于分段循环神经网络的模型，并在西班牙 - 韦萨里卡数据集和德国 - 土耳其数据集的实验中，相较于基准方法有略微提升或相当的表现。