Transformer-based pre-trained language models (PLMs) have dramatically
improved the state of the art in NLP across many tasks. This has led to
substantial interest in analyzing the syntactic knowledge PLMs learn. Previous
approaches to this question have been limited, mostly using test suites or
probes. Here, we propose a novel fully unsupervised parsing approach that
extracts constituency trees from PLM attention heads. We rank transformer
attention heads based on their inherent properties, and create an ensemble of
high-ranking heads to produce the final tree. Our method is adaptable to
low-resource languages, as it does not rely on development sets, which can be
expensive to annotate. Our experiments show that the proposed method often
outperform existing approaches if there is no development set present. Our
unsupervised parser can also be used as a tool to analyze the grammars PLMs
learn implicitly. For this, we use the parse trees induced by our method to
train a neural PCFG and compare it to a grammar derived from a human-annotated
treebank.

本研究使用 Transformer-based pre-trained language models 并提出一种全新的无监督分析方法，通过引入一个高排名头部的集合，用于从语法角度分析 PLMs 的学习知识，从而提取出 PLMs 的句法知识，并用这些知识来训练神经 PCFG，最终与人类注释的语法树库得到比较。

基于自注意力头的无监督成分句法分析

Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads

Many efforts have been devoted to extracting constituency trees from
pre-trained language models, often proceeding in two stages: feature definition
and parsing. However, this kind of methods may suffer from the branching bias
issue, which will inflate the performances on languages with the same branch it
biases to. In this work, we propose quantitatively measuring the branching bias
by comparing the performance gap on a language and its reversed language, which
is agnostic to both language models and extracting methods. Furthermore, we
analyze the impacts of three factors on the branching bias, namely parsing
algorithms, feature definitions, and language models. Experiments show that
several existing works exhibit branching biases, and some implementations of
these three factors can introduce the branching bias.

该研究提出了一种通过比较语言和反向语言的性能差距来量化分支偏差的方法，并分析了分析算法、特征定义和语言模型对分支偏差的影响。实验结果表明，存在一些现有作品存在分支偏差，并且这三个因素的一些实现可能会引入分支偏差。

基于预训练语言模型的句法分支偏差探究

On the Branching Bias of Syntax Extracted from Pre-trained Language  Models

With the recent success and popularity of pre-trained language models (LMs)
in natural language processing, there has been a rise in efforts to understand
their inner workings. In line with such interest, we propose a novel method
that assists us in investigating the extent to which pre-trained LMs capture
the syntactic notion of constituency. Our method provides an effective way of
extracting constituency trees from the pre-trained LMs without training. In
addition, we report intriguing findings in the induced trees, including the
fact that pre-trained LMs outperform other approaches in correctly demarcating
adverb phrases in sentences.

介绍了一种利用预训练语言模型提取短语结构树的方法，并发现预训练语言模型在正确划分句子中的副词短语方面表现优异，从而有助于进一步理解预训练语言模型的内部机制。