In this work, we conduct a detailed analysis on the performance of
legal-oriented pre-trained language models (PLMs). We examine the interplay
between their original objective, acquired knowledge, and legal language
understanding capacities which we define as the upstream, probing, and
downstream performance, respectively. We consider not only the models' size but
also the pre-training corpora used as important dimensions in our study. To
this end, we release a multinational English legal corpus (LeXFiles) and a
legal knowledge probing benchmark (LegalLAMA) to facilitate training and
detailed analysis of legal-oriented PLMs. We release two new legal PLMs trained
on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE. We
find that probing performance strongly correlates with upstream performance in
related legal topics. On the other hand, downstream performance is mainly
driven by the model's size and prior legal knowledge which can be estimated by
upstream and probing performance. Based on these findings, we can conclude that
both dimensions are important for those seeking the development of
domain-specific PLMs.

该研究通过分析基于法律的预训练语言模型的性能，可以得出结论，领域特定的预训练语言模型的开发包括模型的大小和先前的法律知识两个方面。