Neural language models have achieved state-of-the-art performances on many
NLP tasks, and recently have been shown to learn a number of
hierarchically-sensitive syntactic dependencies between individual words.
However, equally important for language processing is the ability to combine
words into phrasal constituents, and use constituent-level features to drive
downstream expectations. Here we investigate neural models' ability to
represent constituent-level features, using coordinated noun phrases as a case
study. We assess whether different neural language models trained on English
and French represent phrase-level number and gender features, and use those
features to drive downstream expectations. Our results suggest that models use
a linear combination of NP constituent number to drive CoordNP/verb number
agreement. This behavior is highly regular and even sensitive to local
syntactic context, however it differs crucially from observed human behavior.
Models have less success with gender agreement. Models trained on large corpora
perform best, and there is no obvious advantage for models trained using
explicit syntactic supervision.

研究了神经语言模型代表短语级别特征的能力，使用协调名词短语作为研究案例，发现模型使用 NP 成分数量的线性组合来驱动 CoordNP/​​verb 数量协议。模型在性别协议方面成功较少。使用大型语料库训练的模型表现最佳，并且使用显式句法监督训练的模型没有明显的优势。

神经语言模型中成分的表征：以协同短语为案例研究

Representation of Constituents in Neural Language Models: Coordination  Phrase as a Case Study

How do neural language models keep track of number agreement between subject
and verb? We show that `diagnostic classifiers', trained to predict number from
the internal states of a language model, provide a detailed understanding of
how, when, and where this information is represented. Moreover, they give us
insight into when and where number information is corrupted in cases where the
language model ends up making agreement errors. To demonstrate the causal role
played by the representations we find, we then use agreement information to
influence the course of the LSTM during the processing of difficult sentences.
Results from such an intervention reveal a large increase in the language
model's accuracy. Together, these results show that diagnostic classifiers give
us an unrivalled detailed look into the representation of linguistic
information in neural models, and demonstrate that this knowledge can be used
to improve their performance.

通过使用诊断分类器来预测内部层状态，我们可以了解神经语言模型如何跟踪主语和动词之间的数的一致性，并发现诊断分类器在提高神经语言模型性能方面的独特作用。

利用诊断分类器研究和改进语言模型跟踪一致性信息的机制

Under the Hood: Using Diagnostic Classifiers to Investigate and Improve  how Language Models Track Agreement Information

Recurrent neural networks (RNNs) have achieved impressive results in a
variety of linguistic processing tasks, suggesting that they can induce
non-trivial properties of language. We investigate here to what extent RNNs
learn to track abstract hierarchical syntactic structure. We test whether RNNs
trained with a generic language modeling objective in four languages (Italian,
English, Hebrew, Russian) can predict long-distance number agreement in various
constructions. We include in our evaluation nonsensical sentences where RNNs
cannot rely on semantic or lexical cues ("The colorless green ideas I ate with
the chair sleep furiously"), and, for Italian, we compare model performance to
human intuitions. Our language-model-trained RNNs make reliable predictions
about long-distance agreement, and do not lag much behind human performance. We
thus bring support to the hypothesis that RNNs are not just shallow-pattern
extractors, but they also acquire deeper grammatical competence.

本文研究了递归神经网络中选择性跟踪语言抽象层级结构的能力，并测试了递归神经网络在四种语言（意大利语、英语、希伯来语、俄语）中使用语言模型目标进行训练是否能够预测各种结构中的长距离数字一致性，结果表明递归神经网络能够可靠地预测长距离一致性而不依赖于词汇或语义信号，支持递归神经网络不仅是浅层模式学习器，而且还能获得较深语法能力的假设。