In this paper, we present a Linguistic Informed Multi-Task BERT (LIMIT-BERT) for learning language representations across multiple linguistic tasks by Multi-Task Learning (MTL). LIMIT-BERT includes five key linguistic syntax and semantics tasks: Part-Of-Speech (POS) tags, constituent and dependency syntactic parsing, span and dependency semantic role labeling (SRL). Besides, LIMIT-BERT adopts linguistics mask strategy: Syntactic and Semantic Phrase Masking which mask all of the tokens corresponding to a syntactic/semantic phrase. Different from recent Multi-Task Deep Neural Networks (MT-DNN) (Liu et al., 2019), our LIMIT-BERT is linguistically motivated and learning in a semi-supervised method which provides large amounts of linguistic-task data as same as BERT learning corpus. As a result, LIMIT-BERT not only improves linguistic tasks performance but also benefits from a regularization effect and linguistic information that leads to more general representations to help adapt to new tasks and domains. LIMIT-BERT obtains new state-of-the-art or competitive results on both span and dependency semantic parsing on Propbank benchmarks and both dependency and constituent syntactic parsing on Penn Treebank.

本文介绍了一种基于Linguistic Informed Multi-Task BERT (LIMIT-BERT)的多任务学习模型，其中包含5个基本的语言句法和语义任务:词性标记、成分和依赖句法分析、跨度和依赖语义角色标记。相比于最近的多任务深度神经网络（MT-DNN），我们的LIMIT-BERT旨在提高语言任务的性能，并通过半监督学习的方式获得大量的语言任务数据，从而提供更普遍的表示来适应新的任务和领域。在Propbank基准测试中，LIMIT-BERT在跨度和依存语义解析上获得了新的最先进或竞争性结果，在Penn Treebank上获得了依存句法和成分句法解析的相应成果。

LIMIT-BERT：语言信息引导的多任务BERT