Scientific experimentation involves an iterative process of creating
hypotheses, designing experiments, running experiments, and analyzing the
results. Can we build AI research agents to perform these long-horizon tasks?
To take a step towards building and evaluating research agents on such
open-ended decision-making tasks, we focus on the problem of machine learning
engineering: given a task description and a dataset, build a high-performing
model. In this paper, we propose MLAgentBench, a suite of ML tasks for
benchmarking AI research agents. Agents can perform actions like
reading/writing files, executing code, and inspecting outputs. With these
actions, agents could run experiments, analyze the results, and modify the code
of entire machine learning pipelines, such as data processing, architecture,
training processes, etc. The benchmark then automatically evaluates the agent's
performance objectively over various metrics related to performance and
efficiency. We also design an LLM-based research agent to automatically perform
experimentation loops in such an environment. Empirically, we find that a
GPT-4-based research agent can feasibly build compelling ML models over many
tasks in MLAgentBench, displaying highly interpretable plans and actions.
However, the success rates vary considerably; they span from almost 90\% on
well-established older datasets to as low as 10\% on recent Kaggle Challenges
-- unavailable during the LLM model's pretraining -- and even 0\% on newer
research challenges like BabyLM. Finally, we identify several key challenges
for LLM-based research agents such as long-term planning and hallucination. Our
code is released at this https URL

我们提出了一种基于 LLM 的研究代理的研究，该代理可以在机器学习工程问题中执行实验循环，并且我们开发了 MLAgentBench 来评估这些代理的性能与效率。