Prior work has shown that, on small amounts of training data, syntactic
neural language models learn structurally sensitive generalisations more
successfully than sequential language models. However, their computational
complexity renders scaling difficult, and it remains an open question whether
structural biases are still necessary when sequential models have access to
ever larger amounts of training data. To answer this question, we introduce an
efficient knowledge distillation (KD) technique that transfers knowledge from a
syntactic language model trained on a small corpus to an LSTM language model,
hence enabling the LSTM to develop a more structurally sensitive representation
of the larger training data it learns from. On targeted syntactic evaluations,
we find that, while sequential LSTMs perform much better than previously
reported, our proposed technique substantially improves on this baseline,
yielding a new state of the art. Our findings and analysis affirm the
importance of structural biases, even in models that learn from large amounts
of data.

利用知识蒸馏技术将小型语法语言模型的知识传递到 LSTM 语言模型，从而使 LSTM 对更大的训练数据开发出更具结构感知性的表示方式。在针对语法的评估中，我们发现，尽管顺序 LSTM 比以前的报告要好得多，但我们提出的技术显著改善了这个基准值，产生了新的最佳成果。我们的发现和分析确认了结构上的偏差的重要性，甚至是在从大量数据中学习的模型中也如此。