Large language models that are capable of zero or few-shot prompting
approaches have given rise to the new research area of prompt engineering.
Recent advances showed that for example Chain-of-Thought (CoT) prompts can
improve arithmetic or common sense tasks significantly. We explore how such
approaches fare with legal reasoning tasks and take the COLIEE entailment task
based on the Japanese Bar exam for testing zero-shot/few-shot and fine-tuning
approaches. Our findings show that while CoT prompting and fine-tuning with
explanations approaches show improvements, the best results are produced by
prompts that are derived from specific legal reasoning techniques such as IRAC
(Issue, Rule, Application, Conclusion). Based on our experiments we improve the
2021 best result from 0.7037 accuracy to 0.8148 accuracy and beat the 2022 best
system of 0.6789 accuracy with an accuracy of 0.7431.

本文探讨了零样本 / 小样本以及微调方法在法律推理任务中的表现，结果发现，源于特定法律推理技术的提示能够获得最佳结果。实验最终将最佳结果从 0.7037 提升至 0.8148，并且准确性高于 2022 最佳系统的 0.6789，为 0.7431。