Realizing the recent advances in Natural Language Processing (NLP) to the
legal sector poses challenging problems such as extremely long sequence
lengths, specialized vocabulary that is usually only understood by legal
professionals, and high amounts of data imbalance. The recent surge of Large
Language Models (LLMs) has begun to provide new opportunities to apply NLP in
the legal domain due to their ability to handle lengthy, complex sequences.
Moreover, the emergence of domain-specific LLMs has displayed extremely
promising results on various tasks. In this study, we aim to quantify how
general LLMs perform in comparison to legal-domain models (be it an LLM or
otherwise). Specifically, we compare the zero-shot performance of three
general-purpose LLMs (ChatGPT-20b, LLaMA-2-70b, and Falcon-180b) on the LEDGAR
subset of the LexGLUE benchmark for contract provision classification. Although
the LLMs were not explicitly trained on legal data, we observe that they are
still able to classify the theme correctly in most cases. However, we find that
their mic-F1/mac-F1 performance is up to 19.2/26.8\% lesser than smaller models
fine-tuned on the legal domain, thus underscoring the need for more powerful
legal-domain LLMs.

近期自然语言处理（NLP）在法律领域的应用面临诸多挑战，包括极长的序列长度、专业词汇仅律师才能理解以及数据不平衡。大型语言模型（LLMs）的出现为 NLP 在法律领域提供了新的机会。本研究旨在量化普通 LLMs 与领域特定模型在法律领域的表现，通过比较三个通用 LLMs（ChatGPT-20b，LLaMA-2-70b 和 Falcon-180b）在 LexGLUE 合同条款分类基准测试集上的零样本性能。尽管 LLMs 未经专门训练法律数据，但我们观察到它们在大多数情况下仍能正确分类主题。然而，我们发现它们的微 F1 / 宏 F1 性能比在法律领域微调的较小模型要低 19.2/26.8％，这凸显了需要更强大的法律领域 LLMs。

大型语言模型的合法性及局限性：构建强大的法律语言模型的论证

Large Language Models are legal but they are not: Making the case for a  powerful LegalLLM

Following the hype around OpenAI's ChatGPT conversational agent, the last
straw in the recent development of Large Language Models (LLMs) that
demonstrate emergent unprecedented zero-shot capabilities, we audit the latest
OpenAI's GPT-3.5 model, `gpt-3.5-turbo', the first available ChatGPT model, in
the LexGLUE benchmark in a zero-shot fashion providing examples in a templated
instruction-following format. The results indicate that ChatGPT achieves an
average micro-F1 score of 47.6% across LexGLUE tasks, surpassing the baseline
guessing rates. Notably, the model performs exceptionally well in some
datasets, achieving micro-F1 scores of 62.8% and 70.2% in the ECtHR B and
LEDGAR datasets, respectively. The code base and model predictions are
available for review on this https URL

研究 OpenAI 的 ChatGPT 对多种语言任务具有零 - shot 性能优势的结果表明，在 LexGLUE 基准测试中，该模型表现良好，平均 micro-F1 得分为 47.6％，特别是在某些数据集中，模型表现非常优秀，如 ECtHR B 和 LEDGAR 数据集中的得分分别达到 62.8％和 70.2％。