In this study, we investigated the potential of ChatGPT, a large language
model developed by OpenAI, for the clinical named entity recognition task
defined in the 2010 i2b2 challenge, in a zero-shot setting with two different
prompt strategies. We compared its performance with GPT-3 in a similar
zero-shot setting, as well as a fine-tuned BioClinicalBERT model using a set of
synthetic clinical notes from MTSamples. Our findings revealed that ChatGPT
outperformed GPT-3 in the zero-shot setting, with F1 scores of 0.418 (vs.0.250)
and 0.620 (vs. 0.480) for exact- and relaxed-matching, respectively. Moreover,
prompts affected ChatGPT's performance greatly, with relaxed-matching F1 scores
of 0.628 vs.0.541 for two different prompt strategies. Although ChatGPT's
performance was still lower than that of the supervised BioClinicalBERT model
(i.e., relaxed-matching F1 scores of 0.628 vs. 0.870), our study demonstrates
the great potential of ChatGPT for clinical NER tasks in a zero-shot setting,
which is much more appealing as it does not require any annotation.

本研究旨在探究以零注释方式进行临床命名实体识别任务中，OpenAI 开发的大型语言模型 ChatGPT 以两种不同提示策略的潜力。我们将其与 GPT-3 在类似的零注释情况下进行比较，以及使用来自 MTSamples 的一组合成临床笔记的精调 BioClinicalBERT 模型。研究发现，与 GPT-3 相比，ChatGPT 在零注释情况下表现优异，并且使用不同的提示策略可以过提高其性能。虽然 ChatGPT 的表现仍低于 BioClinicalBERT 模型，但本研究证明了 ChatGPT 在不需要注释的情况下进行临床 NER 任务的巨大潜力。