In this work, we investigate the potential of large language models (LLMs)
based agents to automate data science tasks, with the goal of comprehending
task requirements, then building and training the best-fit machine learning
models. Despite their widespread success, existing LLM agents are hindered by
generating unreasonable experiment plans within this scenario. To this end, we
present DS-Agent, a novel automatic framework that harnesses LLM agent and
case-based reasoning (CBR). In the development stage, DS-Agent follows the CBR
framework to structure an automatic iteration pipeline, which can flexibly
capitalize on the expert knowledge from Kaggle, and facilitate consistent
performance improvement through the feedback mechanism. Moreover, DS-Agent
implements a low-resource deployment stage with a simplified CBR paradigm to
adapt past successful solutions from the development stage for direct code
generation, significantly reducing the demand on foundational capabilities of
LLMs. Empirically, DS-Agent with GPT-4 achieves an unprecedented 100% success
rate in the development stage, while attaining 36% improvement on average one
pass rate across alternative LLMs in the deployment stage. In both stages,
DS-Agent achieves the best rank in performance, costing \$1.60 and \$0.13 per
run with GPT-4, respectively.

DS-Agent 基于大型语言模型研究数据科学任务自动化，通过利用案例推理框架实现了任务需求理解、机器学习模型构建和训练，并通过低资源部署阶段显著降低了 LLMs 基本能力的要求，从而在性能上取得了最佳排名和显著改进。