We study the fractal structure of language, aiming to provide a precise
formalism for quantifying properties that may have been previously suspected
but not formally shown. We establish that language is: (1) self-similar,
exhibiting complexities at all levels of granularity, with no particular
characteristic context length, and (2) long-range dependent (LRD), with a Hurst
parameter of approximately H=0.70. Based on these findings, we argue that
short-term patterns/dependencies in language, such as in paragraphs, mirror the
patterns/dependencies over larger scopes, like entire documents. This may shed
some light on how next-token prediction can lead to a comprehension of the
structure of text at multiple levels of granularity, from words and clauses to
broader contexts and intents. We also demonstrate that fractal parameters
improve upon perplexity-based bits-per-byte (BPB) in predicting downstream
performance. We hope these findings offer a fresh perspective on language and
the mechanisms underlying the success of LLMs.

我们研究了语言的分形结构，旨在提供一个精确的形式化方法来量化以前可能存在但尚未正式显示的属性。基于我们的发现，我们认为语言具有自相似性，表现出任何粒度级别的复杂性，并且没有特定的上下文长度；并且具有长程依赖性，其 Hurst 参数约为 H = 0.70。我们还证明了分形参数在预测后续表现方面优于困惑度为基础的每字节比特（BPB）。希望这些发现为语言和语言模型的成功背后的机制提供了新的视角。