Pretrained language models are generally acknowledged to be able to encode
syntax [Tenney et al., 2019, Jawahar et al., 2019, Hewitt and Manning, 2019].
In this article, we propose UPOA, an Unsupervised constituent Parsing model
that calculates an Out Association score solely based on the self-attention
weight matrix learned in a pretrained language model as the syntactic distance
for span segmentation. We further propose an enhanced version, UPIO, which
exploits both inside association and outside association scores for estimating
the likelihood of a span. Experiments with UPOA and UPIO disclose that the
linear projection matrices for the query and key in the self-attention
mechanism play an important role in parsing. We therefore extend the
unsupervised models to few-shot parsing models (FPOA, FPIO) that use a few
annotated trees to learn better linear projection matrices for parsing.
Experiments on the Penn Treebank demonstrate that our unsupervised parsing
model UPIO achieves results comparable to the state of the art on short
sentences (length <= 10). Our few-shot parsing model FPIO trained with only 20
annotated trees outperforms a previous few-shot parsing method trained with 50
annotated trees. Experiments on cross-lingual parsing show that both
unsupervised and few-shot parsing methods are better than previous methods on
most languages of SPMRL [Seddah et al., 2013].

本文提出了 UPOA 和 UPIO 两种无监督的成分句法分析模型，其中 UPIO 利用内部关联和外部关联评分估计跨度的可能性，利用少量注释的树来学习更好的线性投影矩阵进行句法分析，实验表明，这两种方法都比之前的方法更有效。

预训练语言模型的自监督和小样本解析

Unsupervised and Few-shot Parsing from Pretrained Language Models

State-of-the-art contextual embeddings are obtained from large language
models available only for a few languages. For others, we need to learn
representations using a multilingual model. There is an ongoing debate on
whether multilingual embeddings can be aligned in a space shared across many
languages. The novel Orthogonal Structural Probe (Limisiewicz and Mare\v{c}ek,
2021) allows us to answer this question for specific linguistic features and
learn a projection based only on mono-lingual annotated datasets. We evaluate
syntactic (UD) and lexical (WordNet) structural information encoded inmBERT's
contextual representations for nine diverse languages. We observe that for
languages closely related to English, no transformation is needed. The
evaluated information is encoded in a shared cross-lingual embedding space. For
other languages, it is beneficial to apply orthogonal transformation learned
separately for each language. We successfully apply our findings to zero-shot
and few-shot cross-lingual parsing.

该研究通过使用新型正交结构探针，针对比较语言特征研究并评估了 9 种不同语言中 mBERT 的语境表示所编码的句法和词汇结构信息，并发现对于与英语密切相关的语言，在跨语言嵌入空间中无需进行转换，而对于其他语言，则需要学习分别进行的正交转换。