Data standardization is a crucial part in data science life cycle. While tools like Pandas offer robust functionalities, their complexity and the manual effort required for customizing code to diverse column types pose significant challenges. Although large language models (LLMs) like ChatGPT have shown promise in automating this process through natural language understanding and code generation, it still demands expert-level programming knowledge and continuous interaction for prompt refinement. To solve these challenges, our key idea is to propose a Python library with declarative, unified APIs for standardizing column types, simplifying the code generation of LLM with concise API calls. We first propose Dataprep.Clean which is written as a component of the Dataprep Library, offers a significant reduction in complexity by enabling the standardization of specific column types with a single line of code. Then we introduce the CleanAgent framework integrating Dataprep.Clean and LLM-based agents to automate the data standardization process. With CleanAgent, data scientists need only provide their requirements once, allowing for a hands-free, automatic standardization process.

我们提出了一个Python库，使用声明性的统一API来标准化列类型，简化LLM的代码生成过程。通过一行代码，我们的库可以显著降低复杂性并实现特定列类型的标准化。而且我们引入了CleanAgent框架，将Dataprep.Clean和基于LLM的agents集成，实现自动化的数据标准化过程。使用CleanAgent，数据科学家只需提供一次要求，即可实现无人工干预的自动标准化过程。

CleanAgent：基于LLM的智能化数据标准化