The MEDIQA-CORR 2024 shared task aims to assess the ability of Large Language
Models (LLMs) to identify and correct medical errors in clinical notes. In this
study, we evaluate the capability of general LLMs, specifically GPT-3.5 and
GPT-4, to identify and correct medical errors with multiple prompting
strategies. Recognising the limitation of LLMs in generating accurate
corrections only via prompting strategies, we propose incorporating error-span
predictions from a smaller, fine-tuned model in two ways: 1) by presenting it
as a hint in the prompt and 2) by framing it as multiple-choice questions from
which the LLM can choose the best correction. We found that our proposed
prompting strategies significantly improve the LLM's ability to generate
corrections. Our best-performing solution with 8-shot + CoT + hints ranked
sixth in the shared task leaderboard. Additionally, our comprehensive analyses
show the impact of the location of the error sentence, the prompted role, and
the position of the multiple-choice option on the accuracy of the LLM. This
prompts further questions about the readiness of LLM to be implemented in
real-world clinical settings.

评估大型语言模型在医疗错误识别和修正方面的能力，提出了多种提示策略来显著提高模型的修正能力，并讨论了错误句子位置、提示角色和选项位置对模型准确性的影响。该研究表明，尽管具有潜力，但是否准备好将大型语言模型应用于真实临床环境仍然需要进一步研究。

MEDIQA-CORR 2024 上的爱丁堡临床自然语言处理：利用提示指导大型语言模型

Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language  Models with Hints

Medical errors in clinical text pose significant risks to patient safety. The
MEDIQA-CORR 2024 shared task focuses on detecting and correcting these errors
across three subtasks: identifying the presence of an error, extracting the
erroneous sentence, and generating a corrected sentence. In this paper, we
present our approach that achieved top performance in all three subtasks. For
the MS dataset, which contains subtle errors, we developed a retrieval-based
system leveraging external medical question-answering datasets. For the UW
dataset, reflecting more realistic clinical notes, we created a pipeline of
modules to detect, localize, and correct errors. Both approaches utilized the
DSPy framework for optimizing prompts and few-shot examples in large language
model (LLM) based programs. Our results demonstrate the effectiveness of LLM
based programs for medical error correction. However, our approach has
limitations in addressing the full diversity of potential errors in medical
documentation. We discuss the implications of our work and highlight future
research directions to advance the robustness and applicability of medical
error detection and correction systems.

医疗文件中的医疗错误对患者安全构成重大风险。本文介绍了我们在三个子任务中均取得顶尖表现的方法，即错误检测、错误句子提取和生成纠正句子。我们利用外部医学问答数据集开发了一套基于检索的系统用于处理微妙错误的 MS 数据集，同时对于反映更真实的临床记录的 UW 数据集，我们创建了一个模块化流程来检测、定位和纠正错误。这两种方法都利用 DSPy 框架来优化大型语言模型程序中的提示和少样本示例。我们的结果表明，基于大型语言模型的程序对于医疗纠错非常有效，但是我们的方法在解决医疗文档中潜在错误的全面多样性方面存在局限性。我们讨论了我们工作的影响，并强调了未来研究方向，以提高医疗错误检测和纠正系统的鲁棒性和适用性。