The disconnect between tokenizer creation and model training in language
models has been known to allow for certain inputs, such as the infamous
SolidGoldMagikarp token, to induce unwanted behaviour. Although such `glitch
tokens' that are present in the tokenizer vocabulary, but are nearly or fully
absent in training, have been observed across a variety of different models, a
consistent way of identifying them has been missing. We present a comprehensive
analysis of Large Language Model (LLM) tokenizers, specifically targeting this
issue of detecting untrained and under-trained tokens. Through a combination of
tokenizer analysis, model weight-based indicators, and prompting techniques, we
develop effective methods for automatically detecting these problematic tokens.
Our findings demonstrate the prevalence of such tokens across various models
and provide insights into improving the efficiency and safety of language
models.

我们通过分析 Tokenizer、基于模型权重的指标和启发式技术的结合，开发了一种有效的方法，用于自动检测在分词器词汇表中存在但在模型训练中很少或完全不存在的问题标记，我们的发现证明了这些标记在各种模型中的普遍存在性，并为改善语言模型的效率和安全性提供了启示。

自动检测大型语言模型中的不充分训练标记

Fishing for Magikarp: Automatically Detecting Under-trained Tokens in  Large Language Models

With the expanding application of Large Language Models (LLMs) in various
domains, it becomes imperative to comprehensively investigate their unforeseen
behaviors and consequent outcomes. In this study, we introduce and
systematically explore the phenomenon of "glitch tokens", which are anomalous
tokens produced by established tokenizers and could potentially compromise the
models' quality of response. Specifically, we experiment on seven top popular
LLMs utilizing three distinct tokenizers and involving a totally of 182,517
tokens. We present categorizations of the identified glitch tokens and symptoms
exhibited by LLMs when interacting with glitch tokens. Based on our observation
that glitch tokens tend to cluster in the embedding space, we propose
GlitchHunter, a novel iterative clustering-based technique, for efficient
glitch token detection. The evaluation shows that our approach notably
outperforms three baseline methods on eight open-source LLMs. To the best of
our knowledge, we present the first comprehensive study on glitch tokens. Our
new detection further provides valuable insights into mitigating
tokenization-related errors in LLMs.

使用三种不同的分词工具，通过实验对七种顶级常用大型语言模型进行研究，发现了异常标志词 “glitch tokens”，并在嵌入空间中发现了它们的聚类规律，通过提出的基于聚类的新技术 GlitchHunter 显著优于三种基准方法，为减少大型语言模型中的分词相关错误提供有价值的洞见。