Automatic medical discovery by AI is a dream of many. One step toward that
goal is to create an AI model to understand clinical studies and synthesize
clinical evidence from the literature. Clinical evidence synthesis currently
relies on systematic reviews of clinical trials and retrospective analyses from
medical literature. However, the rapid expansion of publications presents
challenges in efficiently identifying, summarizing, and updating evidence. We
introduce TrialMind, a generative AI-based pipeline for conducting medical
systematic reviews, encompassing study search, screening, and data extraction
phases. We utilize large language models (LLMs) to drive each pipeline
component while incorporating human expert oversight to minimize errors. To
facilitate evaluation, we also create a benchmark dataset TrialReviewBench, a
custom dataset with 870 annotated clinical studies from 25 meta-analysis papers
across various medical treatments. Our results demonstrate that TrialMind
significantly improves the literature review process, achieving high recall
rates (0.897-1.000) in study searching from over 20 million PubMed studies and
outperforming traditional language model embeddings-based methods in screening
(Recall@20 of 0.227-0.246 vs. 0.000-0.102). Furthermore, our approach surpasses
direct GPT-4 performance in result extraction, with accuracy ranging from 0.65
to 0.84. We also support clinical evidence synthesis in forest plots, as
validated by eight human annotators who preferred TrialMind over the GPT-4
baseline with a winning rate of 62.5%-100% across the involved reviews. Our
findings suggest that an LLM-based clinical evidence synthesis approach, such
as TrialMind, can enable reliable and high-quality clinical evidence synthesis
to improve clinical research efficiency.

利用 TrialMind 等基于 LLMs 的人工智能模型，我们介绍了一种为进行医学系统评价而设计的生成型 AI 流水线。通过结合人类专家的监督，我们证明了 TrialMind 显著提高了文献综述的过程，并在从超过 2000 万篇 PubMed 文章中搜索研究、筛选研究和提取结果的过程中取得了优异的性能。结果还显示，与 GPT-4 基准相比，八位人类评估者更喜欢 TrialMind，其有效率提升了临床研究的效率。