How and to what extent does BERT encode syntactically-sensitive hierarchical
information or positionally-sensitive linear information? Recent work has shown
that contextual representations like BERT perform well on tasks that require
sensitivity to linguistic structure. We present here two studies which aim to
provide a better understanding of the nature of BERT's representations. The
first of these focuses on the identification of structurally-defined elements
using diagnostic classifiers, while the second explores BERT's representation
of subject-verb agreement and anaphor-antecedent dependencies through a
quantitative assessment of self-attention vectors. In both cases, we find that
BERT encodes positional information about word tokens well on its lower layers,
but switches to a hierarchically-oriented encoding on higher layers. We
conclude then that BERT's representations do indeed model linguistically
relevant aspects of hierarchical structure, though they do not appear to show
the sharp sensitivity to hierarchical structure that is found in human
processing of reflexive anaphora.

本文探讨了 BERT 对句法层次结构、位置信息以及自注意力向量的编码，并发现 BERT 在较低的层次上良好地编码了位置信息，在较高的层次上则更倾向于编码层次结构，这表明 BERT 的表示法确实模拟了语言的一些层次结构，并且对于处理反身代词的普遍敏感性没有人类大脑那么强。