Large transformers are powerful architectures for self-supervised analysis of
data of various nature, ranging from protein sequences to text to images. In
these models, the data representation in the hidden layers live in the same
space, and the semantic structure of the dataset emerges by a sequence of
functionally identical transformations between one representation and the next.
We here characterize the geometric and statistical properties of these
representations, focusing on the evolution of such proprieties across the
layers. By analyzing geometric properties such as the intrinsic dimension (ID)
and the neighbor composition we find that the representations evolve in a
strikingly similar manner in transformers trained on protein language tasks and
image reconstruction tasks. In the first layers, the data manifold expands,
becoming high-dimensional, and then it contracts significantly in the
intermediate layers. In the last part of the model, the ID remains
approximately constant or forms a second shallow peak. We show that the
semantic complexity of the dataset emerges at the end of the first peak. This
phenomenon can be observed across many models trained on diverse datasets.
Based on these observations, we suggest using the ID profile as an unsupervised
proxy to identify the layers which are more suitable for downstream learning
tasks.

研究表明，transformers 在蛋白质语言任务和图像重建任务中的表示以非常相似的方式演化，从而呈现出数据流形在隐藏层中的几何和统计特性，而数据流形的语义结构在第一阶段触发。基于这些观察结果，建议使用内在维度作为无监督代理，以确定下游学习任务更合适的层次。