Introduction: With the rapid advances in large language models (LLMs), there
have been numerous new open source as well as commercial models. While recent
publications have explored GPT-4 in its application to extracting information
of interest from radiology reports, there has not been a real-world comparison
of GPT-4 to different leading open-source models.
Materials and Methods: Two different and independent datasets were used. The
first dataset consists of 540 chest x-ray reports that were created at the
Massachusetts General Hospital between July 2019 and July 2021. The second
dataset consists of 500 chest x-ray reports from the ImaGenome dataset. We then
compared the commercial models GPT-3.5 Turbo and GPT-4 from OpenAI to the
open-source models Mistral-7B, Mixtral-8x7B, Llama2-13B, Llama2-70B,
QWEN1.5-72B and CheXbert and CheXpert-labeler in their ability to accurately
label the presence of multiple findings in x-ray text reports using different
prompting techniques.
Results: On the ImaGenome dataset, the best performing open-source model was
Llama2-70B with micro F1-scores of 0.972 and 0.970 for zero- and few-shot
prompts, respectively. GPT-4 achieved micro F1-scores of 0.975 and 0.984,
respectively. On the institutional dataset, the best performing open-source
model was QWEN1.5-72B with micro F1-scores of 0.952 and 0.965 for zero- and
few-shot prompting, respectively. GPT-4 achieved micro F1-scores of 0.975 and
0.973, respectively.
Conclusion: In this paper, we show that while GPT-4 is superior to
open-source models in zero-shot report labeling, the implementation of few-shot
prompting can bring open-source models on par with GPT-4. This shows that
open-source models could be a performant and privacy preserving alternative to
GPT-4 for the task of radiology report classification.

本文展示了尽管 GPT-4 在零样本报告分类方面优于开源模型，但在实现少样本提示的情况下，开源模型与 GPT-4 相当。这表明对于放射学报告分类任务，开源模型可以成为具备高性能和隐私保护的选择替代 GPT-4。

开源软件到位了吗？一项关于商业和开源 LLM 在标注胸部 X 光报告能力方面的对比研究

Is Open-Source There Yet? A Comparative Study on Commercial and  Open-Source LLMs in Their Ability to Label Chest X-Ray Reports

Labelling large datasets for training high-capacity neural networks is a
major obstacle to the development of deep learning-based medical imaging
applications. Here we present a transformer-based network for magnetic
resonance imaging (MRI) radiology report classification which automates this
task by assigning image labels on the basis of free-text expert radiology
reports. Our model's performance is comparable to that of an expert
radiologist, and better than that of an expert physician, demonstrating the
feasibility of this approach. We make code available online for researchers to
label their own MRI datasets for medical imaging applications.

本文介绍了一种基于 Transformer 网络的医学影像分类方法，通过自动标注专家医学报告中的图像标签，克服了大规模数据集标注的困难，其表现相当于专家放射科医师，高于专业医生，该方法可行性很高，并提供了在线代码以帮助研究人员标记自己的 MRI 数据集以供医学应用。