Scoping reviews, a type of literature review, require intensive human effort
to screen large numbers of scholarly sources for their relevance to the review
objectives. This manuscript introduces GPTscreenR, a package for the R
statistical programming language that uses the GPT-4 Large Language Model (LLM)
to automatically screen sources. The package makes use of the chain-of-thought
technique with the goal of maximising performance on complex screening tasks.
In validation against consensus human reviewer decisions, GPTscreenR performed
similarly to an alternative zero-shot technique, with a sensitivity of 71%,
specificity of 89%, and overall accuracy of 84%. Neither method achieved
perfect accuracy nor human levels of intraobserver agreement. GPTscreenR
demonstrates the potential for LLMs to support scholarly work and provides a
user-friendly software framework that can be integrated into existing review
processes.

使用 GPT-4 大型语言模型（LLM）的 GPTscreenR 软件包及链式思维技术，能够自动筛选大量学术资源并在效能和性能方面表现良好，提供了一个可集成到现有审查流程中的用户友好的软件框架。

使用 GPT-4 大型语言模型的自动标题和摘要筛选进行范围审查

Automated title and abstract screening for scoping reviews using the  GPT-4 Large Language Model

Instruction-tuned Large Language Models (It-LLMs) have been exhibiting
outstanding abilities to reason around cognitive states, intentions, and
reactions of all people involved, letting humans guide and comprehend
day-to-day social interactions effectively. In fact, several multiple-choice
questions (MCQ) benchmarks have been proposed to construct solid assessments of
the models' abilities. However, earlier works are demonstrating the presence of
inherent "order bias" in It-LLMs, posing challenges to the appropriate
evaluation. In this paper, we investigate It-LLMs' resilience abilities towards
a series of probing tests using four MCQ benchmarks. Introducing adversarial
examples, we show a significant performance gap, mainly when varying the order
of the choices, which reveals a selection bias and brings into discussion
reasoning abilities. Following a correlation between first positions and model
choices due to positional bias, we hypothesized the presence of structural
heuristics in the decision-making process of the It-LLMs, strengthened by
including significant examples in few-shot scenarios. Finally, by using the
Chain-of-Thought (CoT) technique, we elicit the model to reason and mitigate
the bias by obtaining more robust models.

我们通过使用四个多选题基准来调查 It-LLMs 对多个探测测试的韧性能力。引入对抗性示例，我们展示了在选择顺序变化时出现的显著性能差距，揭示了选择偏差并讨论了推理能力。通过使用连续思维（CoT）技术，我们引发模型进行推理并减轻偏见，从而获得更强大的模型。