Large transformers are powerful architectures for self-supervised analysis of
data of various nature, ranging from protein sequences to text to images. In
these models, the data representation in the hidden layers live in the same
space, and the semantic structure of the dataset emerges by a sequence of
functionally identical transformations between one representation and the next.
We here characterize the geometric and statistical properties of these
representations, focusing on the evolution of such proprieties across the
layers. By analyzing geometric properties such as the intrinsic dimension (ID)
and the neighbor composition we find that the representations evolve in a
strikingly similar manner in transformers trained on protein language tasks and
image reconstruction tasks. In the first layers, the data manifold expands,
becoming high-dimensional, and then it contracts significantly in the
intermediate layers. In the last part of the model, the ID remains
approximately constant or forms a second shallow peak. We show that the
semantic complexity of the dataset emerges at the end of the first peak. This
phenomenon can be observed across many models trained on diverse datasets.
Based on these observations, we suggest using the ID profile as an unsupervised
proxy to identify the layers which are more suitable for downstream learning
tasks.

研究表明，transformers 在蛋白质语言任务和图像重建任务中的表示以非常相似的方式演化，从而呈现出数据流形在隐藏层中的几何和统计特性，而数据流形的语义结构在第一阶段触发。基于这些观察结果，建议使用内在维度作为无监督代理，以确定下游学习任务更合适的层次。

大型 Transformer 模型的隐藏表示的几何学

The geometry of hidden representations of large transformer models

The Gutenberg Literary English Corpus (GLEC, Jacobs, 2018a) provides a rich
source of textual data for research in digital humanities, computational
linguistics or neurocognitive poetics. In this study we address differences
among the different literature categories in GLEC, as well as differences
between authors. We report the results of three studies providing i) topic and
sentiment analyses for six text categories of GLEC (i.e., children and youth,
essays, novels, plays, poems, stories) and its >100 authors, ii) novel measures
of semantic complexity as indices of the literariness, creativity and book
beauty of the works in GLEC (e.g., Jane Austen's six novels), and iii) two
experiments on text classification and authorship recognition using novel
features of semantic complexity. The data on two novel measures estimating a
text's literariness, intratextual variance and stepwise distance (van
Cranenburgh et al., 2019) revealed that plays are the most literary texts in
GLEC, followed by poems and novels. Computation of a novel index of text
creativity (Gray et al., 2016) revealed poems and plays as the most creative
categories with the most creative authors all being poets (Milton, Pope, Keats,
Byron, or Wordsworth). We also computed a novel index of perceived beauty of
verbal art (Kintsch, 2012) for the works in GLEC and predict that Emma is the
theoretically most beautiful of Austen's novels. Finally, we demonstrate that
these novel measures of semantic complexity are important features for text
classification and authorship recognition with overall predictive accuracies in
the range of .75 to .97. Our data pave the way for future computational and
empirical studies of literature or experiments in reading psychology and offer
multiple baselines and benchmarks for analysing and validating other book
corpora.

该研究报告了对 GLEC 中多个文学类别和作者的主题、情感和语义复杂度进行分析的结果，并提出了衡量文学性、创意和文本美感的新指标。同时，他们还提出了这些指标可以作为文本分类和作者识别的重要特征，并通过实验证明了其准确率。

大型英语文学语料库中文本主题、情感、文学性、创造力和美感的计算分析

Computational analyses of the topics, sentiments, literariness, creativity and beauty of texts in a large Corpus of English Literature

Standard evaluations of deep learning models for semantics using naturalistic
corpora are limited in what they can tell us about the fidelity of the learned
representations, because the corpora rarely come with good measures of semantic
complexity. To overcome this limitation, we present a method for generating
data sets of multiply-quantified natural language inference (NLI) examples in
which semantic complexity can be precisely characterized, and we use this
method to show that a variety of common architectures for NLI inevitably fail
to encode crucial information; only a model with forced lexical alignments
avoids this damaging information loss.

本文提出了一种生成数据集的方法，用于生成特定的自然语言推理例子，从而更精确地表征语义复杂性，并指出仅强制词汇对齐的模型可以避免关键信息的损失。