Learning hierarchical structures in sequential data -- from simple
algorithmic patterns to natural language -- in a reliable, generalizable way
remains a challenging problem for neural language models. Past work has shown
that recurrent neural networks (RNNs) struggle to generalize on held-out
algorithmic or syntactic patterns without supervision or some inductive bias.
To remedy this, many papers have explored augmenting RNNs with various
differentiable stacks, by analogy with finite automata and pushdown automata
(PDAs). In this paper, we improve the performance of our recently proposed
Nondeterministic Stack RNN (NS-RNN), which uses a differentiable data structure
that simulates a nondeterministic PDA, with two important changes. First, the
model now assigns unnormalized positive weights instead of probabilities to
stack actions, and we provide an analysis of why this improves training.
Second, the model can directly observe the state of the underlying PDA. Our
model achieves lower cross-entropy than all previous stack RNNs on five
context-free language modeling tasks (within 0.05 nats of the
information-theoretic lower bound), including a task on which the NS-RNN
previously failed to outperform a deterministic stack RNN baseline. Finally, we
propose a restricted version of the NS-RNN that incrementally processes
infinitely long sequences, and we present language modeling results on the Penn
Treebank.

本文介绍了一种基于 PDA 的不确定状态堆栈循环神经网络（NS-RNN）模型，它使用不同 iable stacks，以提高自然语言处理的性能，并提出了一个限制版本的 NS-RNN，以便处理无限长的序列。该模型在五个上下文无关语言建模任务中实现了更低的交叉熵，包括 Penn Treebank 上的一个任务。