In this paper, we propose to study language modelling as a multi-task
problem, bringing together three strands of research: multi-task learning,
linguistics, and interpretability. Based on hypotheses derived from linguistic
theory, we investigate whether language models adhere to learning principles of
multi-task learning during training. To showcase the idea, we analyse the
generalisation behaviour of language models as they learn the linguistic
concept of Negative Polarity Items (NPIs). Our experiments demonstrate that a
multi-task setting naturally emerges within the objective of the more general
task of language modelling.We argue that this insight is valuable for
multi-task learning, linguistics and interpretability research and can lead to
exciting new findings in all three domains.

本文研究语言模型作为多任务问题，结合了多任务学习，语言学和可解释性三个研究领域。通过研究来自语言学理论的假设，我们调查语言模型在训练过程中是否遵循多任务学习的学习原则。 为了展示这个想法，我们分析了语言模型在学习消极极性项（NPIs）的语言概念时的泛化行为。我们的实验表明：在更一般的语言建模任务的目标中，多任务设置自然地出现。我们认为这种洞察力对于多任务学习，语言学和可解释性研究都是有价值的，并且可能导致所有三个领域的令人兴奋的新发现。

多任务问题下的语言建模

Language Modelling as a Multi-Task Problem

In this paper, we attempt to link the inner workings of a neural language
model to linguistic theory, focusing on a complex phenomenon well discussed in
formal linguis- tics: (negative) polarity items. We briefly discuss the leading
hypotheses about the licensing contexts that allow negative polarity items and
evaluate to what extent a neural language model has the ability to correctly
process a subset of such constructions. We show that the model finds a relation
between the licensing context and the negative polarity item and appears to be
aware of the scope of this context, which we extract from a parse tree of the
sentence. With this research, we hope to pave the way for other studies linking
formal linguistics to deep learning.

本文试图将神经语言模型的内部工作与语言学理论联系起来，特别关注了形式语言学中广泛讨论的复杂现象（负）极性项，并评估了神经语言模型处理这种构造的能力。我们展示了模型在许可环境和负极性项之间找到了一种关系，并且似乎意识到了这种环境的范围。通过这项研究，我们希望为其他将形式语言学与深度学习相结合的研究铺平道路。