We evaluated the capability of generative pre-trained transformers~(GPT-4) in
analysis of textual data in tasks that require highly specialized domain
expertise. Specifically, we focused on the task of analyzing court opinions to
interpret legal concepts. We found that GPT-4, prompted with annotation
guidelines, performs on par with well-trained law student annotators. We
observed that, with a relatively minor decrease in performance, GPT-4 can
perform batch predictions leading to significant cost reductions. However,
employing chain-of-thought prompting did not lead to noticeably improved
performance on this task. Further, we demonstrated how to analyze GPT-4's
predictions to identify and mitigate deficiencies in annotation guidelines, and
subsequently improve the performance of the model. Finally, we observed that
the model is quite brittle, as small formatting related changes in the prompt
had a high impact on the predictions. These findings can be leveraged by
researchers and practitioners who engage in semantic/pragmatic annotations of
texts in the context of the tasks requiring highly specialized domain
expertise.

本文评估了生成式预训练转换 (GPT-4) 对高度专业领域的文本数据分析任务的能力，特别关注分析法院裁决以解释法律概念的任务。我们发现，在注释指南的提示下，GPT-4 的表现与经过良好培训的法律学生注释器相当。我们发现，虽然性能略有下降，但 GPT-4 可以进行批量预测，从而降低成本。然而，采用连贯思维提示并没有在此任务上明显提高性能。此外，我们演示了如何分析 GPT-4 的预测，以识别和减轻注释指南的缺陷，进而提高模型的性能。最后，我们观察到该模型非常脆弱，因为提示中的小型格式相关更改对预测结果影响很大。这些发现可被从事高度专业领域任务的文本语义 / 语用注释的研究人员和实践者利用。

GPT-4 在需要高度专业领域知识的文本数据分析任务中的支持能力？

Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly  Specialized Domain Expertise?

With growing amounts of available textual data, development of algorithms
capable of automatic analysis, categorization and summarization of these data
has become a necessity. In this research we present a novel algorithm for
keyword identification, i.e., an extraction of one or multi-word phrases
representing key aspects of a given document, called Transformer-based Neural
Tagger for Keyword IDentification (TNT-KID). By adapting the transformer
architecture for a specific task at hand and leveraging language model
pretraining on a domain specific corpus, the model is capable of overcoming
deficiencies of both supervised and unsupervised state-of-the-art approaches to
keyword extraction by offering competitive and robust performance on a variety
of different datasets while requiring only a fraction of manually labeled data
required by the best performing systems. This study also offers thorough error
analysis with valuable insights into the inner workings of the model and an
ablation study measuring the influence of specific components of the keyword
identification workflow on the overall performance.

本研究提出了一种基于 transformer 框架和领域特定文本预训练的新型关键词提取算法 ——Transformer-based Neural Tagger for Keyword IDentification (TNT-KID), 有效地解决了现有的监督和无监督学习方法在关键词提取上所存在的不足，而且只需要较少的手动标注数据，在多个数据集上达到了较优的表现，并对其内部机制进行了详细的误差分析和关键组件消融实验。