We propose reconstruction probing, a new analysis method for contextualized
representations based on reconstruction probabilities in masked language models
(MLMs). This method relies on comparing the reconstruction probabilities of
tokens in a given sequence when conditioned on the representation of a single
token that has been fully contextualized and when conditioned on only the
decontextualized lexical prior of the model. This comparison can be understood
as quantifying the contribution of contextualization towards reconstruction --
the difference in the reconstruction probabilities can only be attributed to
the representational change of the single token induced by contextualization.
We apply this analysis to three MLMs and find that contextualization boosts
reconstructability of tokens that are close to the token being reconstructed in
terms of linear and syntactic distance. Furthermore, we extend our analysis to
finer-grained decomposition of contextualized representations, and we find that
these boosts are largely attributable to static and positional embeddings at
the input layer.

本文提出了一种新的分析方法 —— 重构探测，基于掩蔽语言模型（MLM）中的重构概率来分析上下文表示。我们应用此分析方法到三个 MLMs，并发现上下文有助于靠近正在重构的标记的标记的可重构性；并且，这些增强主要归于输入层的静态和位置嵌入。