The success of pre-trained contextualized representations has prompted
researchers to analyze them for the presence of linguistic information. Indeed,
it is natural to assume that these pre-trained representations do encode some
level of linguistic knowledge as they have brought about large empirical
improvements on a wide variety of NLP tasks, which suggests they are learning
true linguistic generalization. In this work, we focus on intrinsic probing, an
analysis technique where the goal is not only to identify whether a
representation encodes a linguistic attribute but also to pinpoint where this
attribute is encoded. We propose a novel latent-variable formulation for
constructing intrinsic probes and derive a tractable variational approximation
to the log-likelihood. Our results show that our model is versatile and yields
tighter mutual information estimates than two intrinsic probes previously
proposed in the literature. Finally, we find empirical evidence that
pre-trained representations develop a cross-lingually entangled notion of
morphosyntax.

本文提出了一种新的潜变量公式用于构建内在探测器以确定语言属性所在位置，并提出一个可行的变分逼近方法，用于求解对数似然函数计算，结果表明这个模型能够获得更好的内部探测精度，并且在跨语言的形态句法方面表现良好。

一种隐变量模型用于内部探测

A Latent-Variable Model for Intrinsic Probing

Most modern NLP systems make use of pre-trained contextual representations
that attain astonishingly high performance on a variety of tasks. Such high
performance should not be possible unless some form of linguistic structure
inheres in these representations, and a wealth of research has sprung up on
probing for it. In this paper, we draw a distinction between intrinsic probing,
which examines how linguistic information is structured within a
representation, and the extrinsic probing popular in prior work, which only
argues for the presence of such information by showing that it can be
successfully extracted. To enable intrinsic probing, we propose a novel
framework based on a decomposable multivariate Gaussian probe that allows us to
determine whether the linguistic information in word embeddings is dispersed or
focal. We then probe fastText and BERT for various morphosyntactic attributes
across 36 languages. We find that most attributes are reliably encoded by only
a few neurons, with fastText concentrating its linguistic structure more than
BERT.

本文讨论了自然语言处理系统中之前探测语言结构方法的缺陷，并提出了基于多元高斯探针的内在探测框架，以便于检测词向量的语言信息。通过 36 种语言的实验证明，多数形态语法特征由少数神经元可靠编码，而 fastText 相较于 BERT 更加集中其语言结构。