Large-scale data collection is essential for developing personalized training data, mitigating the shortage of training data, and fine-tuning specialized models. However, creating high-quality datasets quickly and accurately remains a challenge due to annotation errors, the substantial time and costs associated with human labor. To address these issues, we propose Automatic Dataset Construction (ADC), an innovative methodology that automates dataset creation with negligible cost and high efficiency. Taking the image classification task as a starting point, ADC leverages LLMs for the detailed class design and code generation to collect relevant samples via search engines, significantly reducing the need for manual annotation and speeding up the data generation process. Despite these advantages, ADC also encounters real-world challenges such as label errors (label noise) and imbalanced data distributions (label bias). We provide open-source software that incorporates existing methods for label error detection, robust learning under noisy and biased data, ensuring a higher-quality training data and more robust model training procedure. Furthermore, we design three benchmark datasets focused on label noise detection, label noise learning, and class-imbalanced learning. These datasets are vital because there are few existing datasets specifically for label noise detection, despite its importance. Finally, we evaluate the performance of existing popular methods on these datasets, thereby facilitating further research in the field.

本研究解决了在高质量数据集创建中的挑战，包括注释错误和人力成本高昂的问题。提出的自动数据集构建（ADC）方法通过利用大型语言模型实现样本收集和类别设计，大幅提高了数据生成的效率，减少了人工注释的需求。结果展示了该方法在提升训练数据质量和模型训练稳健性方面的潜力，尤其在标签噪声检测和类不平衡学习的基准数据集上具有重要贡献。

自动数据集构建（ADC）：样本收集、数据策划及其应用