Large Language Models (LLMs) could enhance access to the legal system.
However, empirical research on their effectiveness in conducting legal tasks is
scant. We study securities cases involving cryptocurrencies as one of numerous
contexts where AI could support the legal process, studying LLMs' legal
reasoning and drafting capabilities. We examine whether a) an LLM can
accurately determine which laws are potentially being violated from a fact
pattern, and b) whether there is a difference in juror decision-making based on
complaints written by a lawyer compared to an LLM. We feed fact patterns from
real-life cases to GPT-3.5 and evaluate its ability to determine correct
potential violations from the scenario and exclude spurious violations. Second,
we had mock jurors assess complaints written by the LLM and lawyers. GPT-3.5's
legal reasoning skills proved weak, though we expect improvement in future
models, particularly given the violations it suggested tended to be correct (it
merely missed additional, correct violations). GPT-3.5 performed better at
legal drafting, and jurors' decisions were not statistically significantly
associated with the author of the document upon which they based their
decisions. Because LLMs cannot satisfactorily conduct legal reasoning tasks,
they would be unable to replace lawyers at this stage. However, their drafting
skills (though, perhaps, still inferior to lawyers), could provide access to
justice for more individuals by reducing the cost of legal services. Our
research is the first to systematically study LLMs' legal drafting and
reasoning capabilities in litigation, as well as in securities law and
cryptocurrency-related misconduct.

大型语言模型可以提高对法律系统的访问，但是目前很少有关于它们在进行法律任务方面有效性的实证研究。本研究以涉及加密货币的证券案件为背景，研究了语言模型的法律推理和起草能力。我们使用真实案例的情节喂给 GPT-3.5，并评估其确定正确违规行为的能力。研究发现，GPT-3.5 的法律推理能力较弱，但在法律起草方面表现更好。虽然目前无法取代律师，但这些模型的起草能力可以通过降低法律服务成本，为更多人提供司法公正的机会。本研究是第一个系统研究大型语言模型在诉讼、证券法和加密货币相关不当行为中的法律起草和推理能力的研究。