Recent studies have shown that Large Language Models (LLMs) have the
potential to process extremely long text. Many works only evaluate LLMs'
long-text processing ability on the language modeling task, with perplexity
(PPL) as the evaluation metric. However, in our study, we find that there is no
correlation between PPL and LLMs' long-text understanding ability. Besides, PPL
may only reflect the model's ability to model local information instead of
catching long-range dependency. Therefore, only using PPL to prove the model
could process long text is inappropriate. The local focus feature of PPL could
also explain some existing phenomena, such as the great extrapolation ability
of the position method ALiBi. When evaluating a model's ability in long text,
we might pay more attention to PPL's limitation and avoid overly relying on it.

最近的研究表明，大语言模型在处理极长文本方面具有潜力。然而，通过困惑度作为评估指标来评估大语言模型在长文本理解能力方面存在问题，困惑度仅能反映模型对局部信息的建模能力而无法捕捉长距离依赖，因此，仅通过困惑度证明模型可以处理长文本是不合适的。在评估模型的长文本能力时，应当更加注意困惑度的局限性并避免过度依赖。