Lately, propelled by the phenomenal advances around the transformer
architecture, the legal NLP field has enjoyed spectacular growth. To measure
progress, well curated and challenging benchmarks are crucial. However, most
benchmarks are English only and in legal NLP specifically there is no
multilingual benchmark available yet. Additionally, many benchmarks are
saturated, with the best models clearly outperforming the best humans and
achieving near perfect scores. We survey the legal NLP literature and select 11
datasets covering 24 languages, creating LEXTREME. To provide a fair
comparison, we propose two aggregate scores, one based on the datasets and one
on the languages. The best baseline (XLM-R large) achieves both a dataset
aggregate score a language aggregate score of 61.3. This indicates that
LEXTREME is still very challenging and leaves ample room for improvement. To
make it easy for researchers and practitioners to use, we release LEXTREME on
huggingface together with all the code required to evaluate models and a public
Weights and Biases project with all the runs.

使用 Transformer 架构的进步，促进了法律 NLP 领域的发展。为了评估其发展，需要有好的基准数据集。但是目前大部分基准数据集只有英文，而且其中缺少多语言数据集。为此，作者选择了 11 个涵盖 24 种语言的数据集，并创建了 LEXTREME。该工作提出了两种聚合得分，一种基于数据集，一种基于语言。最佳基线模型（XLM-R large）的数据集聚合得分和语言聚合得分均为 61.3，表明 LEXTREME 仍具有很高的挑战难度。为了方便研究人员和从业者使用，作者将 LEXTREME 公开在了 huggingface 并提供了所有用于评估模型的代码以及包括所有运行结果的 public Weights and Biases 项目。