Human language understanding operates at multiple levels of granularity
(e.g., words, phrases, and sentences) with increasing levels of abstraction
that can be hierarchically combined. However, existing deep models with stacked
layers do not explicitly model any sort of hierarchical process. This paper
proposes a recursive Transformer model based on differentiable CKY style binary
trees to emulate the composition process. We extend the bidirectional language
model pre-training objective to this architecture, attempting to predict each
word given its left and right abstraction nodes. To scale up our approach, we
also introduce an efficient pruned tree induction algorithm to enable encoding
in just a linear number of composition steps. Experimental results on language
modeling and unsupervised parsing show the effectiveness of our approach.

本文提出了一种基于递归 Transformer 模型的分层过程的方法，并将双向语言模型预训练目标扩展到此体系结构，试图预测每个单词给定其左右抽象节点。实验结果表明，我们的方法在语言建模和无监督解析方面非常有效。