While LLMs have shown impressive capabilities in solving math or coding
problems, the ability to make scientific discoveries remains a distinct
challenge. This paper proposes a "Turing test for an AI scientist" to assess
whether an AI agent can conduct scientific research independently, without
relying on human-generated knowledge. Drawing inspiration from the historical
development of science, we propose seven benchmark tests that evaluate an AI
agent's ability to make groundbreaking discoveries in various scientific
domains. These tests include inferring the heliocentric model from celestial
observations, discovering the laws of motion in a simulated environment,
deriving the differential equation governing vibrating strings, inferring
Maxwell's equations from electrodynamics simulations, inventing numerical
methods for initial value problems, discovering Huffman coding for data
compression, and developing efficient sorting algorithms. To ensure the
validity of these tests, the AI agent is provided with interactive libraries or
datasets specific to each problem, without access to human knowledge that could
potentially contain information about the target discoveries. The ultimate goal
is to create an AI scientist capable of making novel and impactful scientific
discoveries, surpassing the best human experts in their respective fields.
These "Turing tests" serve as intermediate milestones, assessing the AI agent's
ability to make discoveries that were groundbreaking in their time. If an AI
agent can pass the majority of these seven tests, it would indicate significant
progress towards building an AI scientist, paving the way for future
advancements in autonomous scientific discovery. This paper aims to establish a
benchmark for the capabilities of AI in scientific research and to stimulate
further research in this exciting field.

通过提出一个 “AI 科学家的图灵测试” 来评估 AI 代理能否独立地进行科学研究，这篇论文在提供特定问题的交互式库或数据集的前提下，提出了七个评估 AI 代理在各个科学领域中进行突破性发现能力的基准测试。这些基准测试旨在建立 AI 在科学研究中的能力标准，并激发对这个激动人心领域的进一步研究。

AI 科学家的 “图灵测试

"Turing Tests" For An AI Scientist

The future development of an AI scientist, a tool that is capable of
integrating a variety of experimental data and generating testable hypotheses,
holds immense potential. So far, bespoke machine learning models have been
created to specialize in singular scientific tasks, but otherwise lack the
flexibility of a general purpose model. Here, we show that a general purpose
large language model, chatGPT 3.5-turbo, can be fine-tuned to learn the
structural biophysics of DNA. We find that both fine-tuning models to return
chain-of-thought responses and chaining together models fine-tuned for subtasks
have an enhanced ability to analyze and design DNA sequences and their
structures.

聊天 GPT 3.5-turbo 模型通过细调学习 DNA 的结构生物物理学，展示了一个具备集成各种实验数据并产生可验证假设的 AI 科学家工具的潜力。