This paper investigates the information encoded in the embeddings of large
language models (LLMs). We conduct simulations to analyze the representation
entropy and discover a power law relationship with model sizes. Building upon
this observation, we propose a theory based on (conditional) entropy to
elucidate the scaling law phenomenon. Furthermore, we delve into the
auto-regressive structure of LLMs and examine the relationship between the last
token and previous context tokens using information theory and regression
techniques. Specifically, we establish a theoretical connection between the
information gain of new tokens and ridge regression. Additionally, we explore
the effectiveness of Lasso regression in selecting meaningful tokens, which
sometimes outperforms the closely related attention weights. Finally, we
conduct controlled experiments, and find that information is distributed across
tokens, rather than being concentrated in specific "meaningful" tokens alone.

该研究探讨了大型语言模型（LLMs）中嵌入的信息编码，并发现与模型大小存在幂律关系的表示熵。基于此观察，提出了一个基于（条件）熵的理论以阐明该缩放定律现象。此外，通过使用信息论和回归技术，研究 LLMs 的自回归结构，并检查最后一个标记与前文标记之间的关系。具体地，我们建立了新标记的信息增益与岭回归之间的理论联系。此外，我们还探索了 Lasso 回归在选择有意义的标记方面的有效性，有时优于相关的注意力权重。最后，通过进行对比实验，发现信息分布在各个标记中，而不仅仅集中在特定的 “有意义” 标记中。