It is generally thought that transformer-based large language models benefit
from pre-training by learning generic linguistic knowledge that can be focused
on a specific task during fine-tuning. However, we propose that much of the
benefit from pre-training may be captured by geometric characteristics of the
latent space representations, divorced from any specific linguistic knowledge.
In this work we explore the relationship between GLUE benchmarking task
performance and a variety of measures applied to the latent space resulting
from BERT-type contextual language models. We find that there is a strong
linear relationship between a measure of quantized cell density and average
GLUE performance and that these measures may be predictive of otherwise
surprising GLUE performance for several non-standard BERT-type models from the
literature. These results may be suggestive of a strategy for decreasing
pre-training requirements, wherein model initialization can be informed by the
geometric characteristics of the model's latent space.

我们提出了一个新的方法，通过研究 BERT 类型模型的潜在空间的几何特征，来探索预训练对 GLUE 基准任务表现的影响，并发现潜在空间的量化细胞密度与 GLUE 性能有着强烈的线性关系，这些结果表明可以减少预训练要求，通过模型潜在空间的几何特征来初始化模型。