Recently CKY-based models show great potential in unsupervised grammar
induction thanks to their human-like encoding paradigm, which runs recursively
and hierarchically, but requires $O(n^3)$ time-complexity. Recursive
Transformer based on Differentiable Trees (R2D2) makes it possible to scale to
large language model pre-training even with complex tree encoder by introducing
a heuristic pruning method. However, the rule-based pruning approach suffers
from local optimum and slow inference issues. In this paper, we fix those
issues in a unified method. We propose to use a top-down parser as a
model-based pruning method, which also enables parallel encoding during
inference. Typically, our parser casts parsing as a split point scoring task,
which first scores all split points for a given sentence, and then recursively
splits a span into two by picking a split point with the highest score in the
current span. The reverse order of the splits is considered as the order of
pruning in R2D2 encoder. Beside the bi-directional language model loss, we also
optimize the parser by minimizing the KL distance between tree probabilities
from parser and R2D2. Our experiments show that our Fast-R2D2 improves
performance significantly in grammar induction and achieves competitive results
in downstream classification tasks.

本文提出一个使用自上而下分析的模型进行基于规则剪枝的自适应语法归纳的方法，使得像 CKY 模型这样的基于递归的解析器可以更加高效地处理复杂的自然语言，实验证明该方法在语法归纳方面表现显著。