We evaluated the capability of generative pre-trained transformers~(GPT-4) in
analysis of textual data in tasks that require highly specialized domain
expertise. Specifically, we focused on the task of analyzing court opinions to
interpret legal concepts. We found that GPT-4, prompted with annotation
guidelines, performs on par with well-trained law student annotators. We
observed that, with a relatively minor decrease in performance, GPT-4 can
perform batch predictions leading to significant cost reductions. However,
employing chain-of-thought prompting did not lead to noticeably improved
performance on this task. Further, we demonstrated how to analyze GPT-4's
predictions to identify and mitigate deficiencies in annotation guidelines, and
subsequently improve the performance of the model. Finally, we observed that
the model is quite brittle, as small formatting related changes in the prompt
had a high impact on the predictions. These findings can be leveraged by
researchers and practitioners who engage in semantic/pragmatic annotations of
texts in the context of the tasks requiring highly specialized domain
expertise.

本文评估了生成式预训练转换 (GPT-4) 对高度专业领域的文本数据分析任务的能力，特别关注分析法院裁决以解释法律概念的任务。我们发现，在注释指南的提示下，GPT-4 的表现与经过良好培训的法律学生注释器相当。我们发现，虽然性能略有下降，但 GPT-4 可以进行批量预测，从而降低成本。然而，采用连贯思维提示并没有在此任务上明显提高性能。此外，我们演示了如何分析 GPT-4 的预测，以识别和减轻注释指南的缺陷，进而提高模型的性能。最后，我们观察到该模型非常脆弱，因为提示中的小型格式相关更改对预测结果影响很大。这些发现可被从事高度专业领域任务的文本语义 / 语用注释的研究人员和实践者利用。