Recent latent tree learning models can learn constituency parsing without any
exposure to human-annotated tree structures. One such model is ON-LSTM (Shen et
al., 2019), which is trained on language modelling and has
near-state-of-the-art performance on unsupervised parsing. In order to better
understand the performance and consistency of the model as well as how the
parses it generates are different from gold-standard PTB parses, we replicate
the model with different restarts and examine their parses. We find that (1)
the model has reasonably consistent parsing behaviors across different
restarts, (2) the model struggles with the internal structures of complex noun
phrases, (3) the model has a tendency to overestimate the height of the split
points right before verbs. We speculate that both problems could potentially be
solved by adopting a different training task other than unidirectional language
modelling.

研究表明，使用 ON-LSTM 等模型进行自然语言处理，可以在没有人为注释的情况下学习组成句法分析，相对最佳状态的性能表现较好，但在解析内部结构更复杂的名词短语时存在困难，并可能高估在动词之前分割点的高度。我们推测采用不同于单向语言建模的训练任务可能有助于解决这两个问题。

使用有序神经元实现潜在树学习：其生成哪些句法结构？

Latent Tree Learning with Ordered Neurons: What Parses Does It Produce?

Natural language is hierarchically structured: smaller units (e.g., phrases)
are nested within larger units (e.g., clauses). When a larger constituent ends,
all of the smaller constituents that are nested within it must also be closed.
While the standard LSTM architecture allows different neurons to track
information at different time scales, it does not have an explicit bias towards
modeling a hierarchy of constituents. This paper proposes to add such an
inductive bias by ordering the neurons; a vector of master input and forget
gates ensures that when a given neuron is updated, all the neurons that follow
it in the ordering are also updated. Our novel recurrent architecture, ordered
neurons LSTM (ON-LSTM), achieves good performance on four different tasks:
language modeling, unsupervised parsing, targeted syntactic evaluation, and
logical inference.

本文提出了一种新的循环神经网络体系结构，称为 ON-LSTM，通过对神经元进行排序来添加合理的生物学归纳偏差，从而增强了语言建模、无监督解析、有针对性的句法评估和逻辑推理等四项任务的性能表现。