Legal tasks and datasets are often used as benchmarks for the capabilities of
language models. However, openly available annotated datasets are rare. In this
paper, we introduce AGB-DE, a corpus of 3,764 clauses from German consumer
contracts that have been annotated and legally assessed by legal experts.
Together with the data, we present a first baseline for the task of detecting
potentially void clauses, comparing the performance of an SVM baseline with
three fine-tuned open language models and the performance of GPT-3.5. Our
results show the challenging nature of the task, with no approach exceeding an
F1-score of 0.54. While the fine-tuned models often performed better with
regard to precision, GPT-3.5 outperformed the other approaches with regard to
recall. An analysis of the errors indicates that one of the main challenges
could be the correct interpretation of complex clauses, rather than the
decision boundaries of what is permissible and what is not.

介绍了一个包含 3764 个德国消费者合同从业条款的 AGB-DE 语料库，并对检测潜在无效从句的任务进行了首次基线评估，结果显示该任务的挑战性，并指出理解复杂从句是其中的主要挑战之一。