Screening prioritisation in medical systematic reviews aims to rank the set
of documents retrieved by complex Boolean queries. The goal is to prioritise
the most important documents so that subsequent review steps can be carried out
more efficiently and effectively. The current state of the art uses the final
title of the review to rank documents using BERT-based neural neural rankers.
However, the final title is only formulated at the end of the review process,
which makes this approach impractical as it relies on ex post facto
information. At the time of screening, only a rough working title is available,
with which the BERT-based ranker achieves is significantly worse than the final
title. In this paper, we explore alternative sources of queries for screening
prioritisation, such as the Boolean query used to retrieve the set of documents
to be screened, and queries generated by instruction-based generative large
language models such as ChatGPT and Alpaca. Our best approach is not only
practical based on the information available at screening time, but is similar
in effectiveness with the final title.

筛选优先级是医学系统性审查中的一个重要环节，本研究探索了用于优先级排序的替代查询来源，如用于检索待筛选文档集的布尔查询和基于指令的生成型大型语言模型，我们的最佳方法在筛选时基于可用信息，且与最终标题的有效性相似。

为实现更有效的系统性回顾筛选优先级而生成自然语言查询

Generating Natural Language Queries for More Effective Systematic Review  Screening Prioritisation

Medical systematic reviews can be very costly and resource intensive. We
explore how Large Language Models (LLMs) can support and be trained to perform
literature screening when provided with a detailed set of selection criteria.
Specifically, we instruction tune LLaMA and Guanaco models to perform abstract
screening for medical systematic reviews. Our best model, Bio-SIEVE,
outperforms both ChatGPT and trained traditional approaches, and generalises
better across medical domains. However, there remains the challenge of adapting
the model to safety-first scenarios. We also explore the impact of multi-task
training with Bio-SIEVE-Multi, including tasks such as PICO extraction and
exclusion reasoning, but find that it is unable to match single-task
Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards
specialising LLMs for the biomedical systematic review process and explore its
future developmental opportunities. We release our models, code and a list of
DOIs to reconstruct our dataset for reproducibility.

通过对详细的选择标准进行训练，我们使用大型语言模型（LLMs）来支持和执行医学系统评价文献筛选。我们的最佳模型 Bio-SIEVE 在医学领域中优于 ChatGPT 和传统方法，并在多个医学领域中具有更好的泛化性能。然而，将该模型适应安全优先的情境仍然是一个挑战。我们还探讨了 Bio-SIEVE-Multi 的多任务训练的影响，包括 PICO 提取和排除推理等任务，但发现它无法与单任务的 Bio-SIEVE 的表现相匹配。我们认为 Bio-SIEVE 是将 LLMs 专门用于生物医学系统评价过程的重要一步，并探讨了它的未来发展机会。我们公开发布了我们的模型、代码和 DOI 列表，以便重现我们的数据集。

Bio-SIEVE: 探索调整大型语言模型以实现系统性综述自动化

Bio-SIEVE: Exploring Instruction Tuning Large Language Models for  Systematic Review Automation

Medical systematic reviews are crucial for informing clinical decision making
and healthcare policy. But producing such reviews is onerous and
time-consuming. Thus, high-quality evidence synopses are not available for many
questions and may be outdated even when they are available. Large language
models (LLMs) are now capable of generating long-form texts, suggesting the
tantalizing possibility of automatically generating literature reviews on
demand. However, LLMs sometimes generate inaccurate (and potentially
misleading) texts by hallucinating or omitting important information. In the
healthcare context, this may render LLMs unusable at best and dangerous at
worst. Most discussion surrounding the benefits and risks of LLMs have been
divorced from specific applications. In this work, we seek to qualitatively
characterize the potential utility and risks of LLMs for assisting in
production of medical evidence reviews. We conducted 16 semi-structured
interviews with international experts in systematic reviews, grounding
discussion in the context of generating evidence reviews. Domain experts
indicated that LLMs could aid writing reviews, as a tool for drafting or
creating plain language summaries, generating templates or suggestions,
distilling information, crosschecking, and synthesizing or interpreting text
inputs. But they also identified issues with model outputs and expressed
concerns about potential downstream harms of confidently composed but
inaccurate LLM outputs which might mislead. Other anticipated potential
downstream harms included lessened accountability and proliferation of
automatically generated reviews that might be of low quality. Informed by this
qualitative analysis, we identify criteria for rigorous evaluation of
biomedical LLMs aligned with domain expert views.

该研究考虑了使用大型自然语言处理模型辅助生成医学证据综述的益处和风险，以 16 个半结构化面谈为基础，为领域讨论提出了评估标准。