Large Language Models (LLMs) have shown remarkable capabilities in language
understanding and generation. Nonetheless, it was also witnessed that LLMs tend
to produce inaccurate responses to specific queries. This deficiency can be
traced to the tokenization step LLMs must undergo, which is an inevitable
limitation inherent to all LLMs. In fact, incorrect tokenization is the
critical point that hinders LLMs in understanding the input precisely, thus
leading to unsatisfactory output. To demonstrate this flaw of LLMs, we
construct an adversarial dataset, named as $\textbf{ADT (Adversarial Dataset
for Tokenizer)}$, which draws upon the vocabularies of various open-source LLMs
to challenge LLMs' tokenization. ADT consists of two subsets: the manually
constructed ADT-Human and the automatically generated ADT-Auto. Our empirical
results reveal that our ADT is highly effective on challenging the tokenization
of leading LLMs, including GPT-4o, Llama-3, Qwen2.5-max and so on, thus
degrading these LLMs' capabilities. Moreover, our method of automatic data
generation has been proven efficient and robust, which can be applied to any
open-source LLMs. To the best of our knowledge, our study is the first to
investigating LLMs' vulnerability in terms of challenging their token
segmentation, which will shed light on the subsequent research of improving
LLMs' capabilities through optimizing their tokenization process and
algorithms.

基于大型语言模型的令牌划分漏洞的研究，提出了对模型进行攻击的对抗性数据集（ADT），通过挑战语言模型对输入的划分，揭示令牌划分对模型性能的重要影响，并为改善语言模型能力提供优化划分过程和算法的研究方法。

令牌化的重要性！通过挑战其令牌化降低大型语言模型的性能

Tokenization Matters! Degrading Large Language Models through  Challenging Their Tokenization

Neural Theory-of-Mind (N-ToM), machine's ability to understand and keep track
of the mental states of others, is pivotal in developing socially intelligent
agents. However, prevalent N-ToM benchmarks have several shortcomings,
including the presence of ambiguous and artificial narratives, absence of
personality traits and preferences, a lack of questions addressing characters'
psychological mental states, and limited diversity in the questions posed. In
response to these issues, we construct OpenToM, a new benchmark for assessing
N-ToM with (1) longer and clearer narrative stories, (2) characters with
explicit personality traits, (3) actions that are triggered by character
intentions, and (4) questions designed to challenge LLMs' capabilities of
modeling characters' mental states of both the physical and psychological
world. Using OpenToM, we reveal that state-of-the-art LLMs thrive at modeling
certain aspects of mental states in the physical world but fall short when
tracking characters' mental states in the psychological world.

机器学习中的神经心智理论 (N-ToM) 是理解和跟踪他人心理状态的关键，本研究构建了一个新的基准 OpenToM，通过长而清晰的叙述、具有明确人格特征的角色以及挑战人工智能模型对心理和心理世界中角色心理状态建模能力的设计提出了一些问题，揭示了最先进的模型在物理世界中对心理状态的建模方面表现出色，但在心理世界中的心理状态跟踪方面表现不足。