In this paper, we introduce our approaches using Transformer-based models for
different problems of the COLIEE 2021 automatic legal text processing
competition. Automated processing of legal documents is a challenging task
because of the characteristics of legal documents as well as the limitation of
the amount of data. With our detailed experiments, we found that
Transformer-based pretrained language models can perform well with automated
legal text processing problems with appropriate approaches. We describe in
detail the processing steps for each task such as problem formulation, data
processing and augmentation, pretraining, finetuning. In addition, we introduce
to the community two pretrained models that take advantage of parallel
translations in legal domain, NFSP and NMSP. In which, NFSP achieves the
state-of-the-art result in Task 5 of the competition. Although the paper
focuses on technical reporting, the novelty of its approaches can also be an
useful reference in automated legal document processing using Transformer-based
models.

本文介绍一种基于 Transformer 模型的方法，以解决 COLIEE 2021 自动法律文本处理竞赛中的不同问题，并详细描述了每个任务的处理步骤，包括问题公式化、数据处理和增强、预训练和微调。此外，本文还介绍了两个在法律领域利用平行翻译的预训练模型 NFSP 和 NMSP，其中 NFSP 在该竞赛的任务 5 中取得了最优结果。

基于 Transformer 的法律文本处理方法

Transformer-based Approaches for Legal Text Processing

There has been mounting evidence that pretrained language models fine-tuned
on large and diverse supervised datasets can transfer well to a variety of
out-of-domain tasks. In this work, we investigate this transfer ability to the
legal domain. For that, we participated in the legal case entailment task of
COLIEE 2021, in which we use such models with no adaptations to the target
domain. Our submissions achieved the highest scores, surpassing the second-best
team by more than six percentage points. Our experiments confirm a
counter-intuitive result in the new paradigm of pretrained language models:
given limited labeled data, models with little or no adaptation to the target
task can be more robust to changes in the data distribution than models
fine-tuned on it. Code is available at this https URL.

本研究通过参与 COLIEE 2021 的法律案例蕴含任务，考察了预训练语言模型在法律领域的迁移能力。实验结果表明，在有限标记数据的情况下，与适应目标任务的 Fine-tuning 模型相比，没有或只部分适应的模型对数据分布的更改更加稳健。