Adverse event (AE) extraction following COVID-19 vaccines from text data is crucial for monitoring and analyzing the safety profiles of immunizations. Traditional deep learning models are adept at learning intricate feature representations and dependencies in sequential data, but often require extensive labeled data. In contrast, large language models (LLMs) excel in understanding contextual information, but exhibit unstable performance on named entity recognition tasks, possibly due to their broad but unspecific training. This study aims to evaluate the effectiveness of LLMs and traditional deep learning models in AE extraction, and to assess the impact of ensembling these models on performance. In this study, we utilized reports and posts from the VAERS (n=621), Twitter (n=9,133), and Reddit (n=131) as our corpora. Our goal was to extract three types of entities: "vaccine", "shot", and "ae". We explored and fine-tuned (except GPT-4) multiple LLMs, including GPT-2, GPT-3.5, GPT-4, and Llama-2, as well as traditional deep learning models like RNN and BioBERT. To enhance performance, we created ensembles of the three models with the best performance. For evaluation, we used strict and relaxed F1 scores to evaluate the performance for each entity type, and micro-average F1 was used to assess the overall performance. The ensemble model achieved the highest performance in "vaccine", "shot", and "ae" with strict F1-scores of 0.878, 0.930, and 0.925, respectively, along with a micro-average score of 0.903. In conclusion, this study demonstrates the effectiveness and robustness of ensembling fine-tuned traditional deep learning models and LLMs, for extracting AE-related information. This study contributes to the advancement of biomedical natural language processing, providing valuable insights into improving AE extraction from text data for pharmacovigilance and public health surveillance.

本篇研究旨在评估LLMs和传统深度学习模型在提取AE信息方面的有效性，并评估这些模型集成对性能的影响。研究发现，利用优化后的传统深度学习模型和LLMs进行集成，能够在提取疫苗、注射和不良事件方面实现最高性能，同时对整体性能的评估也得到了显著提高。这项研究对生物医学自然语言处理的发展做出了贡献，并为从文本数据中提取AE相关信息的药物监测和公共卫生监测提供了宝贵的见解。

Improving Entity Recognition Using Ensembles of Deep Learning and
  Fine-tuned Large Language Models: A Case Study on Adverse Event Extraction
  from Multiple Sources

通过深度学习和精调的大型语言模型构建集成模型以提升实体识别性能：多源不良事件抽取案例研究