Data is one of the most critical elements in building a large language model.
However, existing systems either fail to customize a corpus curation pipeline
or neglect to leverage comprehensive corpus assessment for iterative
optimization of the curation. To this end, we present a pretraining corpus
curation and assessment platform called Oasis -- a one-stop system for data
quality improvement and quantification with user-friendly interactive
interfaces. Specifically, the interactive modular rule filter module can devise
customized rules according to explicit feedback. The debiased neural filter
module builds the quality classification dataset in a negative-centric manner
to remove the undesired bias. The adaptive document deduplication module could
execute large-scale deduplication with limited memory resources. These three
parts constitute the customized data curation module. And in the holistic data
assessment module, a corpus can be assessed in local and global views, with
three evaluation means including human, GPT-4, and heuristic metrics. We
exhibit a complete process to use Oasis for the curation and assessment of
pretraining data. In addition, an 800GB bilingual corpus curated by Oasis is
publicly released.

数据是构建大规模语言模型的最关键元素之一，我们提出了一个名为 Oasis 的预训练数据整理和评估平台，通过用户友好的交互界面实现数据质量改进和量化评估，平台包括自定义数据整理模块和全面的数据评估模块，还发布了由 Oasis 整理的一个 800GB 双语语料库。