Background: Transformer-based language models have shown strong performance
on many Natural LanguageProcessing (NLP) tasks. Masked Language Models (MLMs)
attract sustained interest because they can be adaptedto different languages
and sub-domains through training or fine-tuning on specific corpora while
remaining lighterthan modern Large Language Models (LLMs). Recently, several
MLMs have been released for the biomedicaldomain in French, and experiments
suggest that they outperform standard French counterparts. However,
nosystematic evaluation comparing all models on the same corpora is available.
Objective: This paper presentsan evaluation of masked language models for
biomedical French on the task of clinical named entity recognition.Material and
methods: We evaluate biomedical models CamemBERT-bio and DrBERT and compare
them tostandard French models CamemBERT, FlauBERT and FrALBERT as well as
multilingual mBERT using three publicallyavailable corpora for clinical named
entity recognition in French. The evaluation set-up relies on
gold-standardcorpora as released by the corpus developers. Results: Results
suggest that CamemBERT-bio outperformsDrBERT consistently while FlauBERT offers
competitive performance and FrAlBERT achieves the lowest carbonfootprint.
Conclusion: This is the first benchmark evaluation of biomedical masked
language models for Frenchclinical entity recognition that compares model
performance consistently on nested entity recognition using metricscovering
performance and environmental impact.

这篇论文是关于对医学法语遮蔽语言模型在临床命名实体识别任务上的评估研究，发现 CamemBERT-bio 在性能上表现优于 DrBERT，而 FlauBERT 则具有竞争力，FrALBERT 的碳排放量最低。这是首个基准评估了法语医学遮蔽语言模型在临床实体识别方面的性能，并比较了其在嵌套实体识别上的性能和环境影响。

基于法语的临床命名实体识别性能评估

A Benchmark Evaluation of Clinical Named Entity Recognition in French

In this study, we investigated the potential of ChatGPT, a large language
model developed by OpenAI, for the clinical named entity recognition task
defined in the 2010 i2b2 challenge, in a zero-shot setting with two different
prompt strategies. We compared its performance with GPT-3 in a similar
zero-shot setting, as well as a fine-tuned BioClinicalBERT model using a set of
synthetic clinical notes from MTSamples. Our findings revealed that ChatGPT
outperformed GPT-3 in the zero-shot setting, with F1 scores of 0.418 (vs.0.250)
and 0.620 (vs. 0.480) for exact- and relaxed-matching, respectively. Moreover,
prompts affected ChatGPT's performance greatly, with relaxed-matching F1 scores
of 0.628 vs.0.541 for two different prompt strategies. Although ChatGPT's
performance was still lower than that of the supervised BioClinicalBERT model
(i.e., relaxed-matching F1 scores of 0.628 vs. 0.870), our study demonstrates
the great potential of ChatGPT for clinical NER tasks in a zero-shot setting,
which is much more appealing as it does not require any annotation.

本研究旨在探究以零注释方式进行临床命名实体识别任务中，OpenAI 开发的大型语言模型 ChatGPT 以两种不同提示策略的潜力。我们将其与 GPT-3 在类似的零注释情况下进行比较，以及使用来自 MTSamples 的一组合成临床笔记的精调 BioClinicalBERT 模型。研究发现，与 GPT-3 相比，ChatGPT 在零注释情况下表现优异，并且使用不同的提示策略可以过提高其性能。虽然 ChatGPT 的表现仍低于 BioClinicalBERT 模型，但本研究证明了 ChatGPT 在不需要注释的情况下进行临床 NER 任务的巨大潜力。