Incorporating syntax into neural approaches in NLP has a multitude of
practical and scientific benefits. For instance, a language model that is
syntax-aware is likely to be able to produce better samples; even a
discriminative model like BERT with a syntax module could be used for core NLP
tasks like unsupervised syntactic parsing. Rapid progress in recent years was
arguably spurred on by the empirical success of the Parsing-Reading-Predict
architecture of (Shen et al., 2018a), later simplified by the Order Neuron LSTM
of (Shen et al., 2019). Most notably, this is the first time neural approaches
were able to successfully perform unsupervised syntactic parsing (evaluated by
various metrics like F-1 score).
However, even heuristic (much less fully mathematical) understanding of why
and when these architectures work is lagging severely behind. In this work, we
answer representational questions raised by the architectures in (Shen et al.,
2018a, 2019), as well as some transition-based syntax-aware language models
(Dyer et al., 2016): what kind of syntactic structure can current neural
approaches to syntax represent? Concretely, we ground this question in the
sandbox of probabilistic context-free-grammars (PCFGs), and identify a key
aspect of the representational power of these approaches: the amount and
directionality of context that the predictor has access to when forced to make
parsing decision. We show that with limited context (either bounded, or
unidirectional), there are PCFGs, for which these approaches cannot represent
the max-likelihood parse; conversely, if the context is unlimited, they can
represent the max-likelihood parse of any PCFG.

本文探讨了当前神经语法学习方法对句法结构的表征方式，以及限制神经语法学习方法的表征能力的主要因素，并利用概率上下文无关文法为基础，说明上下文对神经语法学习方法进行语法分析决策时的关键作用。