In the era of flourishing large-scale models, the challenge of selecting and
optimizing datasets from the vast and complex sea of data, to enhance the
performance of large language models within the constraints of limited
computational resources, has become paramount. This paper details our solution
for the BetterMixture challenge, which focuses on the fine-tuning data mixing
for large language models. Our approach, which secured third place,
incorporates data deduplication, low-level and high-level quality filtering,
and diversity selection. The foundation of our solution is Ke-Data-Juicer, an
extension of Data-Juicer, demonstrating its robust capabilities in handling and
optimizing data for large language models.

在大规模模型时代，选择和优化海量复杂数据集以提高大型语言模型的性能，在有限的计算资源约束下成为一项重大挑战。本文详细介绍了我们在 BetterMixture 挑战中的解决方案，着重于对大型语言模型进行微调数据混合。我们的方法在数据去重、低级和高级质量过滤以及多样性选择方面表现出色，在处理和优化大型语言模型的数据方面，基于 Data-Juicer 的扩展工具 Ke-Data-Juicer 使其具备了强大的能力。

技术报告：更好混合的竞争解决方案

Technical Report: Competition Solution For BetterMixture

The immense evolution in Large Language Models (LLMs) has underscored the
importance of massive, diverse, and high-quality data. Despite this, existing
open-source tools for LLM data processing remain limited and mostly tailored to
specific datasets, with an emphasis on the reproducibility of released data
over adaptability and usability, inhibiting potential applications. In
response, we propose a one-stop, powerful yet flexible and user-friendly LLM
data processing system named Data-Juicer. Our system offers over 50 built-in
versatile operators and pluggable tools, which synergize modularity,
composability, and extensibility dedicated to diverse LLM data processing
needs. By incorporating visualized and automatic evaluation capabilities,
Data-Juicer enables a timely feedback loop to accelerate data processing and
gain data insights. To enhance usability, Data-Juicer provides out-of-the-box
components for users with various backgrounds, and fruitful data recipes for
LLM pre-training and post-tuning usages. Further, we employ multi-facet system
optimization and seamlessly integrate Data-Juicer with both LLM and distributed
computing ecosystems, to enable efficient and scalable data processing.
Empirical validation of the generated data recipes reveals considerable
improvements in LLaMA performance for various pre-training and post-tuning
cases, demonstrating up to 7.45% relative improvement of averaged score across
16 LLM benchmarks and 16.25% higher win rate using pair-wise GPT-4 evaluation.
The system's efficiency and scalability are also validated, supported by up to
88.7% reduction in single-machine processing time, 77.1% and 73.1% less memory
and CPU usage respectively, and 7.91x processing acceleration when utilizing
distributed computing ecosystems. Our system, data recipes, and multiple
tutorial demos are released, calling for broader research centered on LLM data.

我们提出了一种强大而灵活、易于使用的数据处理系统 Data-Juicer，它提供了 50 多种内置的多功能操作符和可插拔工具，通过可视化和自动化评估能力，加快数据处理并获得数据洞察力，提供用户友好的界面并发布多种数据配方，同时实现高效和可扩展的数据处理。