State-of-the-art LSTM language models trained on large corpora learn
sequential contingencies in impressive detail and have been shown to acquire a
number of non-local grammatical dependencies with some success. Here we
investigate whether supervision with hierarchical structure enhances learning
of a range of grammatical dependencies, a question that has previously been
addressed only for subject-verb agreement. Using controlled experimental
methods from psycholinguistics, we compare the performance of word-based LSTM
models versus two models that represent hierarchical structure and deploy it in
left-to-right processing: Recurrent Neural Network Grammars (RNNGs) (Dyer et
al., 2016) and a incrementalized version of the Parsing-as-Language-Modeling
configuration from Chariak et al., (2016). Models are tested on a diverse range
of configurations for two classes of non-local grammatical dependencies in
English---Negative Polarity licensing and Filler--Gap Dependencies. Using the
same training data across models, we find that structurally-supervised models
outperform the LSTM, with the RNNG demonstrating best results on both types of
grammatical dependencies and even learning many of the Island Constraints on
the filler--gap dependency. Structural supervision thus provides data
efficiency advantages over purely string-based training of neural language
models in acquiring human-like generalizations about non-local grammatical
dependencies.

本研究探究了基于分层结构预训练的神经语言模型与 LSTM 模型在学习语音上的差异，并发现基于分层结构的猝死神经网络语法模型具有明显优势。

结构化监督改善非本地语法依赖关系的学习

Structural Supervision Improves Learning of Non-Local Grammatical  Dependencies

There has been relatively little attention to incorporating linguistic prior
to neural machine translation. Much of the previous work was further
constrained to considering linguistic prior on the source side. In this paper,
we propose a hybrid model, called NMT+RNNG, that learns to parse and translate
by combining the recurrent neural network grammar into the attention-based
neural machine translation. Our approach encourages the neural machine
translation model to incorporate linguistic prior during training, and lets it
translate on its own afterward. Extensive experiments with four language pairs
show the effectiveness of the proposed NMT+RNNG.

本文提出了一种名为 NMT+RNNG 的混合模型，它将循环神经网络语法与基于注意力的神经机器翻译相结合，通过训练鼓励神经机器翻译模型吸收语言先验知识，并在其后进行自主翻译。4 种语言配对的实验表明该模型非常有效。