The immense evolution in Large Language Models (LLMs) has underscored the
importance of massive, diverse, and high-quality data. Despite this, existing
open-source tools for LLM data processing remain limited and mostly tailored to
specific datasets, with an emphasis on the reproducibility of released data
over adaptability and usability, inhibiting potential applications. In
response, we propose a one-stop, powerful yet flexible and user-friendly LLM
data processing system named Data-Juicer. Our system offers over 50 built-in
versatile operators and pluggable tools, which synergize modularity,
composability, and extensibility dedicated to diverse LLM data processing
needs. By incorporating visualized and automatic evaluation capabilities,
Data-Juicer enables a timely feedback loop to accelerate data processing and
gain data insights. To enhance usability, Data-Juicer provides out-of-the-box
components for users with various backgrounds, and fruitful data recipes for
LLM pre-training and post-tuning usages. Further, we employ multi-facet system
optimization and seamlessly integrate Data-Juicer with both LLM and distributed
computing ecosystems, to enable efficient and scalable data processing.
Empirical validation of the generated data recipes reveals considerable
improvements in LLaMA performance for various pre-training and post-tuning
cases, demonstrating up to 7.45% relative improvement of averaged score across
16 LLM benchmarks and 16.25% higher win rate using pair-wise GPT-4 evaluation.
The system's efficiency and scalability are also validated, supported by up to
88.7% reduction in single-machine processing time, 77.1% and 73.1% less memory
and CPU usage respectively, and 7.91x processing acceleration when utilizing
distributed computing ecosystems. Our system, data recipes, and multiple
tutorial demos are released, calling for broader research centered on LLM data.

我们提出了一种强大而灵活、易于使用的数据处理系统 Data-Juicer，它提供了 50 多种内置的多功能操作符和可插拔工具，通过可视化和自动化评估能力，加快数据处理并获得数据洞察力，提供用户友好的界面并发布多种数据配方，同时实现高效和可扩展的数据处理。