In this paper, we investigate to which extent contextual neural language
models (LMs) implicitly learn syntactic structure. More concretely, we focus on
constituent structure as represented in the Penn Treebank (PTB). Using standard
probing techniques based on diagnostic classifiers, we assess the accuracy of
representing constituents of different categories within the neuron activations
of a LM such as RoBERTa. In order to make sure that our probe focuses on
syntactic knowledge and not on implicit semantic generalizations, we also
experiment on a PTB version that is obtained by randomly replacing constituents
with each other while keeping syntactic structure, i.e., a semantically
ill-formed but syntactically well-formed version of the PTB. We find that 4
pretrained transfomer LMs obtain high performance on our probing tasks even on
manipulated data, suggesting that semantic and syntactic knowledge in their
representations can be separated and that constituency information is in fact
learned by the LM. Moreover, we show that a complete constituency tree can be
linearly separated from LM representations.

本文研究神经语言模型对句法结构的学习能力，发现预训练的 Transfomer LM 即使在语义有错的情况下仍能准确地代表不同类别的成分，并且语法信息确实被 LM 学习并与语义信息分离。