Representation of linguistic phenomena in computational language models is
typically assessed against the predictions of existing linguistic theories of
these phenomena. Using the notion of polarity as a case study, we show that
this is not always the most adequate set-up. We probe polarity via so-called
'negative polarity items' (in particular, English 'any') in two pre-trained
Transformer-based models (BERT and GPT-2). We show that - at least for polarity
- metrics derived from language models are more consistent with data from
psycholinguistic experiments than linguistic theory predictions. Establishing
this allows us to more adequately evaluate the performance of language models
and also to use language models to discover new insights into natural language
grammar beyond existing linguistic theories. This work contributes to
establishing closer ties between psycholinguistic experiments and experiments
with language models.

本文以极性为案例，使用预训练的 Transformer 模型（BERT 和 GPT-2）探讨了负极性项目（特别是英语中的 'any'）的极性，并证明了语言模型派生的指标比语言学理论预测更符合心理语言学实验数据。这可以更充分地评估语言模型的性能，并使用语言模型来发现自然语言语法方面的新见解。该研究有助于加强心理语言学实验和语言模型实验之间的紧密联系。

循环中的转换器：语言神经模型中的极性

Transformers in the loop: Polarity in neural models of language

There are two major classes of natural language grammar -- the dependency
grammar that models one-to-one correspondences between words and the
constituency grammar that models the assembly of one or several corresponded
words. While previous unsupervised parsing methods mostly focus on only
inducing one class of grammars, we introduce a novel model, StructFormer, that
can simultaneously induce dependency and constituency structure. To achieve
this, we propose a new parsing framework that can jointly generate a
constituency tree and dependency graph. Then we integrate the induced
dependency relations into the transformer, in a differentiable manner, through
a novel dependency-constrained self-attention mechanism. Experimental results
show that our model can achieve strong results on unsupervised constituency
parsing, unsupervised dependency parsing, and masked language modeling at the
same time.

本文提出了一种新的自然语言无监督语法分析模型，StructFormer，可以同时识别依存结构和组成结构，通过引入一个新的解析框架，该框架可以一起生成组成树和依赖图，并通过新颖的依赖约束的自我关注机制将依赖关系集成到 transformer 中，在无监督组成句法分析、无监督依存分析和蒙特卡罗语言建模方面取得了强大的结果。