A growing body of literature has focused on detailing the linguistic
knowledge embedded in large, pretrained language models. Existing work has
shown that non-linguistic biases in models can drive model behavior away from
linguistic generalizations. We hypothesized that competing linguistic processes
within a language, rather than just non-linguistic model biases, could obscure
underlying linguistic knowledge. We tested this claim by exploring a single
phenomenon in four languages: English, Chinese, Spanish, and Italian. While
human behavior has been found to be similar across languages, we find
cross-linguistic variation in model behavior. We show that competing processes
in a language act as constraints on model behavior and demonstrate that
targeted fine-tuning can re-weight the learned constraints, uncovering
otherwise dormant linguistic knowledge in models. Our results suggest that
models need to learn both the linguistic constraints in a language and their
relative ranking, with mismatches in either producing non-human-like behavior.

本文研究了大型预训练语言模型中内嵌的语言知识，并探索了同时存在的竞争性语言过程如何影响模型行为。通过对四种语言中的一个现象进行实验，发现模型行为存在跨语言差异。而有针对性的微调可以重新建立学习的约束，揭示模型中否则未表现出的语言知识。

通过有针对性的微调揭示神经模型中基于约束的行为

Uncovering Constraint-Based Behavior in Neural Models via Targeted  Fine-Tuning

A standard approach to evaluating language models analyzes how models assign
probabilities to valid versus invalid syntactic constructions (i.e. is a
grammatical sentence more probable than an ungrammatical sentence). Our work
uses ambiguous relative clause attachment to extend such evaluations to cases
of multiple simultaneous valid interpretations, where stark grammaticality
differences are absent. We compare model performance in English and Spanish to
show that non-linguistic biases in RNN LMs advantageously overlap with
syntactic structure in English but not Spanish. Thus, English models may appear
to acquire human-like syntactic preferences, while models trained on Spanish
fail to acquire comparable human-like preferences. We conclude by relating
these results to broader concerns about the relationship between comprehension
(i.e. typical language model use cases) and production (which generates the
training data for language models), suggesting that necessary linguistic biases
are not present in the training signal at all.

通过使用模糊性相对子句附加来扩展语言模型在多重同时有效解释的情况下的评估方法。我们比较了英语和西班牙语模型的表现，以展示 RNN LMs 中的非语言偏见优势重叠在英语而非西班牙语的句法结构上。因此，英语模型可能具有类似于人类的句法偏好，而在西班牙语上训练的模型无法获得类似的人类句法偏好。