Transformer-based pre-trained language models (PLMs) have dramatically
improved the state of the art in NLP across many tasks. This has led to
substantial interest in analyzing the syntactic knowledge PLMs learn. Previous
approaches to this question have been limited, mostly using test suites or
probes. Here, we propose a novel fully unsupervised parsing approach that
extracts constituency trees from PLM attention heads. We rank transformer
attention heads based on their inherent properties, and create an ensemble of
high-ranking heads to produce the final tree. Our method is adaptable to
low-resource languages, as it does not rely on development sets, which can be
expensive to annotate. Our experiments show that the proposed method often
outperform existing approaches if there is no development set present. Our
unsupervised parser can also be used as a tool to analyze the grammars PLMs
learn implicitly. For this, we use the parse trees induced by our method to
train a neural PCFG and compare it to a grammar derived from a human-annotated
treebank.

本研究使用 Transformer-based pre-trained language models 并提出一种全新的无监督分析方法，通过引入一个高排名头部的集合，用于从语法角度分析 PLMs 的学习知识，从而提取出 PLMs 的句法知识，并用这些知识来训练神经 PCFG，最终与人类注释的语法树库得到比较。