We propose a neural language model capable of unsupervised syntactic
structure induction. The model leverages the structure information to form
better semantic representations and better language modeling. Standard
recurrent neural networks are limited by their structure and fail to
efficiently use syntactic information. On the other hand, tree-structured
recursive networks usually require additional structural supervision at the
cost of human expert annotation. In this paper, We propose a novel neural
language model, called the Parsing-Reading-Predict Networks (PRPN), that can
simultaneously induce the syntactic structure from unannotated sentences and
leverage the inferred structure to learn a better language model. In our model,
the gradient can be directly back-propagated from the language model loss into
the neural parsing network. Experiments show that the proposed model can
discover the underlying syntactic structure and achieve state-of-the-art
performance on word/character-level language model tasks.

本文提出了一种新型的神经语言模型 Parsing-Reading-Predict Networks（PRPN），利用其特定的神经网络结构能够自动识别未标注的句子的句法结构，并利用它来学习更好的语言模型。实验证明，该模型能够发现底层的句法结构，并在单词 / 字符水平的语言模型任务上取得了最先进水平。