Pre-trained language encoders -- functions that represent text as vectors --
are an integral component of many NLP tasks. We tackle a natural question in
language encoder analysis: What does it mean for two encoders to be similar? We
contend that a faithful measure of similarity needs to be \emph{intrinsic},
that is, task-independent, yet still be informative of \emph{extrinsic}
similarity -- the performance on downstream tasks. It is common to consider two
encoders similar if they are \emph{homotopic}, i.e., if they can be aligned
through some transformation. In this spirit, we study the properties of
\emph{affine} alignment of language encoders and its implications on extrinsic
similarity. We find that while affine alignment is fundamentally an asymmetric
notion of similarity, it is still informative of extrinsic similarity. We
confirm this on datasets of natural language representations. Beyond providing
useful bounds on extrinsic similarity, affine intrinsic similarity also allows
us to begin uncovering the structure of the space of pre-trained encoders by
defining an order over them.

预训练语言编码器的相似性及其对下游任务性能的影响进行了分析，发现了仿射对齐的特性以及其对外在相似度的启示，从而揭示了预训练编码器空间的结构。

关于语言编码器间的仿射同伦

On Affine Homotopy between Language Encoders

We conduct a thorough study to diagnose the behaviors of pre-trained language
encoders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical
errors. Specifically, we collect real grammatical errors from non-native
speakers and conduct adversarial attacks to simulate these errors on clean text
data. We use this approach to facilitate debugging models on downstream
applications. Results confirm that the performance of all tested models is
affected but the degree of impact varies. To interpret model behaviors, we
further design a linguistic acceptability task to reveal their abilities in
identifying ungrammatical sentences and the position of errors. We find that
fixed contextual encoders with a simple classifier trained on the prediction of
sentence correctness are able to locate error positions. We also design a cloze
test for BERT and discover that BERT captures the interaction between errors
and specific tokens in context. Our results shed light on understanding the
robustness and behaviors of language encoders against grammatical errors.

本文旨在研究预训练语言编码器（ELMo，BERT 和 RoBERTa）在面对自然语法错误时的表现，通过采集真实语法错误和进行对抗性攻击来模拟这些错误对干净文本数据的影响。结果证实，所有测试模型的性能都受到了影响，但影响程度有所不同。此外，我们设计了一个语言接受度任务来揭示它们在识别不符合语法的句子和错误位置方面的能力。本文的结果有助于理解语言编码器对语法错误的鲁棒性和行为。

关于语言编码器对语法错误的鲁棒性

On the Robustness of Language Encoders against Grammatical Errors

Even as pre-trained language encoders such as BERT are shared across many
tasks, the output layers of question answering, text classification, and
regression models are significantly different. Span decoders are frequently
used for question answering, fixed-class, classification layers for text
classification, and similarity-scoring layers for regression tasks, We show
that this distinction is not necessary and that all three can be unified as
span extraction. A unified, span-extraction approach leads to superior or
comparable performance in supplementary supervised pre-trained, low-data, and
multi-task learning experiments on several question answering, text
classification, and regression benchmarks.

本论文的研究表明，使用统一的 span extraction 方法可以在多个任务中取得优越或可比的性能，而不必采用不同的输出层（如 span decoder 和 fixed-class classification layers 等）来适应不同的任务，这可应用在多种领域，如预训练语言编码、少数据学习和多任务学习等。