Realizing the recent advances in Natural Language Processing (NLP) to the
legal sector poses challenging problems such as extremely long sequence
lengths, specialized vocabulary that is usually only understood by legal
professionals, and high amounts of data imbalance. The recent surge of Large
Language Models (LLMs) has begun to provide new opportunities to apply NLP in
the legal domain due to their ability to handle lengthy, complex sequences.
Moreover, the emergence of domain-specific LLMs has displayed extremely
promising results on various tasks. In this study, we aim to quantify how
general LLMs perform in comparison to legal-domain models (be it an LLM or
otherwise). Specifically, we compare the zero-shot performance of three
general-purpose LLMs (ChatGPT-20b, LLaMA-2-70b, and Falcon-180b) on the LEDGAR
subset of the LexGLUE benchmark for contract provision classification. Although
the LLMs were not explicitly trained on legal data, we observe that they are
still able to classify the theme correctly in most cases. However, we find that
their mic-F1/mac-F1 performance is up to 19.2/26.8\% lesser than smaller models
fine-tuned on the legal domain, thus underscoring the need for more powerful
legal-domain LLMs.

近期自然语言处理（NLP）在法律领域的应用面临诸多挑战，包括极长的序列长度、专业词汇仅律师才能理解以及数据不平衡。大型语言模型（LLMs）的出现为 NLP 在法律领域提供了新的机会。本研究旨在量化普通 LLMs 与领域特定模型在法律领域的表现，通过比较三个通用 LLMs（ChatGPT-20b，LLaMA-2-70b 和 Falcon-180b）在 LexGLUE 合同条款分类基准测试集上的零样本性能。尽管 LLMs 未经专门训练法律数据，但我们观察到它们在大多数情况下仍能正确分类主题。然而，我们发现它们的微 F1 / 宏 F1 性能比在法律领域微调的较小模型要低 19.2/26.8％，这凸显了需要更强大的法律领域 LLMs。