For a language model (LM) to faithfully model human language, it must
compress vast, potentially infinite information into relatively few dimensions.
We propose analyzing compression in (pre-trained) LMs from two points of view:
geometric and information-theoretic. We demonstrate that the two views are
highly correlated, such that the intrinsic geometric dimension of linguistic
data predicts their coding length under the LM. We then show that, in turn,
high compression of a linguistic dataset predicts rapid adaptation to that
dataset, confirming that being able to compress linguistic information is an
important part of successful LM performance. As a practical byproduct of our
analysis, we evaluate a battery of intrinsic dimension estimators for the first
time on linguistic data, showing that only some encapsulate the relationship
between information-theoretic compression, geometric compression, and
ease-of-adaptation.

通过分析语言模型（LM）中的压缩方法，从几何和信息论的角度，我们证明这两种视角高度相关，表明语言数据的内在几何维度可以预测其在 LM 下的编码长度，进而表明语言信息压缩能力是成功 LM 性能的重要组成部分。同时，我们还评估了一系列首次应用于语言数据的特征维度估计器，表明只有其中的一部分能够概括信息论压缩、几何压缩和适应性关系。