There are two major classes of natural language grammars -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded words. While previous unsupervised parsing methods mostly focus on only inducing one class of grammars, we introduce a novel model, StructFormer, that can induce dependency and constituency structure at the same time. To achieve this, we propose a new parsing framework that can jointly generate a constituency tree and dependency graph. Then we integrate the induced dependency relations into the transformer, in a differentiable manner, through a novel dependency-constrained self-attention mechanism. Experimental results show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time.

本文提出了一种新的自然语言无监督语法分析模型，StructFormer，可以同时识别依存结构和组成结构，通过引入一个新的解析框架，该框架可以一起生成组成树和依赖图，并通过新颖的依赖约束的自我关注机制将依赖关系集成到 transformer 中，在无监督组成句法分析、无监督依存分析和蒙特卡罗语言建模方面取得了强大的结果。

StructFormer: 联合无监督遮蔽语言建模归纳依存结构和组成结构