We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data is generated using a massive probabilistic grammar (based on state-split PCFGs), that is itself derived from a large natural language corpus, but also provides us complete control over the generative process. We describe and release both grammar and corpus, and test for the naturalness of our generated data. This approach allows us to define closed-form expressions to efficiently compute exact lower bounds on obtainable perplexity using both causal and masked language modelling. Our results show striking differences between neural language modelling architectures and training objectives in how closely they allow approximating the lower bound on perplexity. Our approach also allows us to directly compare learned representations to symbolic rules in the underlying source. We experiment with various techniques for interpreting model behaviour and learning dynamics. With access to the underlying true source, our results show striking differences and outcomes in learning dynamics between different classes of words.

我们提出了一种使用人工制造的类似语言的数据来训练、评估和解释神经语言模型的设置。使用大规模概率语法（基于状态拆分的PCFGs）生成数据，该语法源自大型自然语言语料库，但也可完全控制生成过程。通过比较神经语言建模架构和训练目标对可获取困惑度下界的逼近程度的差异，我们得出了显著的结果。我们的方法还允许直接将学到的表示与底层源中的符号规则进行比较，并尝试了各种用于解释模型行为和学习动态的技术。通过访问底层真实源，我们的结果显示出不同类别的单词在学习动态方面存在明显的差异和结果。

透明、来源：评估和解释有真实分布访问的语言模型