While self-supervised learning has made rapid advances in natural language
processing, it remains unclear when researchers should engage in
resource-intensive domain-specific pretraining (domain pretraining). The law,
puzzlingly, has yielded few documented instances of substantial gains to domain
pretraining in spite of the fact that legal language is widely seen to be
unique. We hypothesize that these existing results stem from the fact that
existing legal NLP tasks are too easy and fail to meet conditions for when
domain pretraining can help. To address this, we first present CaseHOLD (Case
Holdings On Legal Decisions), a new dataset comprised of over 53,000+ multiple
choice questions to identify the relevant holding of a cited case. This dataset
presents a fundamental task to lawyers and is both legally meaningful and
difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second,
we assess performance gains on CaseHOLD and existing legal NLP datasets. While
a Transformer architecture (BERT) pretrained on a general corpus (Google Books
and Wikipedia) improves performance, domain pretraining (using corpus of
approximately 3.5M decisions across all courts in the U.S. that is larger than
BERT's) with a custom legal vocabulary exhibits the most substantial
performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12%
improvement on BERT) and consistent performance gains across two other legal
tasks. Third, we show that domain pretraining may be warranted when the task
exhibits sufficient similarity to the pretraining corpus: the level of
performance increase in three legal tasks was directly tied to the domain
specificity of the task. Our findings inform when researchers should engage
resource-intensive pretraining and show that Transformer-based architectures,
too, learn embeddings suggestive of distinct legal language.

本文介绍了一个新的 NLP 数据集 CaseHOLD，用于测试法律方面的 NLP 任务，同时指出在某些特定的领域语料的预训练条件下，Transformer-based architectures 的性能可能比一般的预训练模型更优秀，我们的发现有助于指导研究人员何时需要进行域预训练。