The ability to automatically generate accurate protocols for scientific
experiments would represent a major step towards the automation of science.
Large Language Models (LLMs) have impressive capabilities on a wide range of
tasks, such as question answering and the generation of coherent text and code.
However, LLMs can struggle with multi-step problems and long-term planning,
which are crucial for designing scientific experiments. Moreover, evaluation of
the accuracy of scientific protocols is challenging, because experiments can be
described correctly in many different ways, require expert knowledge to
evaluate, and cannot usually be executed automatically. Here we present an
automatic evaluation framework for the task of planning experimental protocols,
and we introduce BioProt: a dataset of biology protocols with corresponding
pseudocode representations. To measure performance on generating scientific
protocols, we use an LLM to convert a natural language protocol into
pseudocode, and then evaluate an LLM's ability to reconstruct the pseudocode
from a high-level description and a list of admissible pseudocode functions. We
evaluate GPT-3 and GPT-4 on this task and explore their robustness. We
externally validate the utility of pseudocode representations of text by
generating accurate novel protocols using retrieved pseudocode, and we run a
generated protocol successfully in our biological laboratory. Our framework is
extensible to the evaluation and improvement of language model planning
abilities in other areas of science or other areas that lack automatic
evaluation.

科学实验的自动生成协议，通过使用基于伪代码表示的生物协议数据集进行评估，并评估 GPT-3 和 GPT-4 在生成科学协议方面的能力和鲁棒性。

BioPlanner：自动评估生物学中协议规划的 LLMs

BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology

Transformer-based large language models are rapidly advancing in the field of
machine learning research, with applications spanning natural language,
biology, chemistry, and computer programming. Extreme scaling and reinforcement
learning from human feedback have significantly improved the quality of
generated text, enabling these models to perform various tasks and reason about
their choices. In this paper, we present an Intelligent Agent system that
combines multiple large language models for autonomous design, planning, and
execution of scientific experiments. We showcase the Agent's scientific
research capabilities with three distinct examples, with the most complex being
the successful performance of catalyzed cross-coupling reactions. Finally, we
discuss the safety implications of such systems and propose measures to prevent
their misuse.

本论文展示了一个结合多个大型语言模型的智能代理系统，可自主设计、规划和执行科学实验，并通过三个不同的实例展示代理的科学研究能力，最为复杂的是成功执行加催化交叉偶联反应。最后，讨论了这种系统的安全影响，并提出了防止滥用的措施。