With the advent of large language models (LLMs), there has been growing
interest in exploring their potential for medical applications. This research
aims to investigate the ability of LLMs, specifically ChatGPT, in the context
of pharmacovigilance event extraction, of which the main goal is to identify
and extract adverse events or potential therapeutic events from textual medical
sources. We conduct extensive experiments to assess the performance of ChatGPT
in the pharmacovigilance event extraction task, employing various prompts and
demonstration selection strategies. The findings demonstrate that while ChatGPT
demonstrates reasonable performance with appropriate demonstration selection
strategies, it still falls short compared to fully fine-tuned small models.
Additionally, we explore the potential of leveraging ChatGPT for data
augmentation. However, our investigation reveals that the inclusion of
synthesized data into fine-tuning may lead to a decrease in performance,
possibly attributed to noise in the ChatGPT-generated labels. To mitigate this,
we explore different filtering strategies and find that, with the proper
approach, more stable performance can be achieved, although constant
improvement remains elusive.

探索大型语言模型 (LLMs) 在医学应用方面的潜力，特别是在药物监测事件提取方面的 ChatGPT 的能力。我们进行了广泛的实验，评估了 ChatGPT 在药物监测事件提取任务中的性能，并采用了各种提示和演示选择策略。研究结果表明，虽然 ChatGPT 在适当的演示选择策略下表现出了合理的性能，但与完全微调的小型模型相比仍然有所不足。此外，我们探索了利用 ChatGPT 进行数据增强的潜力。然而，我们的研究发现将合成数据引入微调可能会导致性能下降，可能是由于 ChatGPT 生成的标签中存在噪声。为了减轻这个问题，我们探索了不同的过滤策略，并发现在适当的方法下可以实现更稳定的性能，尽管不断改进仍然具有挑战性。