In this work, we conduct a detailed analysis on the performance of
legal-oriented pre-trained language models (PLMs). We examine the interplay
between their original objective, acquired knowledge, and legal language
understanding capacities which we define as the upstream, probing, and
downstream performance, respectively. We consider not only the models' size but
also the pre-training corpora used as important dimensions in our study. To
this end, we release a multinational English legal corpus (LeXFiles) and a
legal knowledge probing benchmark (LegalLAMA) to facilitate training and
detailed analysis of legal-oriented PLMs. We release two new legal PLMs trained
on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE. We
find that probing performance strongly correlates with upstream performance in
related legal topics. On the other hand, downstream performance is mainly
driven by the model's size and prior legal knowledge which can be estimated by
upstream and probing performance. Based on these findings, we can conclude that
both dimensions are important for those seeking the development of
domain-specific PLMs.

该研究通过分析基于法律的预训练语言模型的性能，可以得出结论，领域特定的预训练语言模型的开发包括模型的大小和先前的法律知识两个方面。

LeXFiles 和 LegalLAMA: 促进英语跨国法律语言模型的发展

LeXFiles and LegalLAMA: Facilitating English Multinational Legal  Language Model Development

Recently, domain-specific PLMs have been proposed to boost the task
performance of specific domains (e.g., biomedical and computer science) by
continuing to pre-train general PLMs with domain-specific corpora. However,
this Domain-Adaptive Pre-Training (DAPT; Gururangan et al. (2020)) tends to
forget the previous general knowledge acquired by general PLMs, which leads to
a catastrophic forgetting phenomenon and sub-optimal performance. To alleviate
this problem, we propose a new framework of General Memory Augmented
Pre-trained Language Model (G-MAP), which augments the domain-specific PLM by a
memory representation built from the frozen general PLM without losing any
general knowledge. Specifically, we propose a new memory-augmented layer, and
based on it, different augmented strategies are explored to build the memory
representation and then adaptively fuse it into the domain-specific PLM. We
demonstrate the effectiveness of G-MAP on various domains (biomedical and
computer science publications, news, and reviews) and different kinds (text
classification, QA, NER) of tasks, and the extensive results show that the
proposed G-MAP can achieve SOTA results on all tasks.

提出了一种 General Memory Augmented Pre-trained Language Model (G-MAP) 框架，通过从未失去通用知识的冻结通用 PLM 构建内存表示来增广专业领域 PLM，并探索不同的增广策略来将其自适应地融合到专业领域 PLM 中，证明了 G-MAP 对各种类型的任务，在各种领域中可以实现 SOTA 结果。