The performance of a large language model (LLM) depends heavily on the
quality and size of its pretraining dataset. However, the pretraining datasets
for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly
available and very little is known about how they were created. In this work,
we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl
snapshots that produces better-performing LLMs than other open pretraining
datasets. To advance the understanding of how best to curate high-quality
pretraining datasets, we carefully document and ablate all of the design
choices used in FineWeb, including in-depth investigations of deduplication and
filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion
token collection of educational text filtered from FineWeb. LLMs pretrained on
FineWeb-Edu exhibit dramatically better performance on knowledge- and
reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we
publicly release our data curation codebase and all of the models trained
during our ablation experiments.

FineWeb 是一个从 96 个 Common Crawl 快照中派生出的 1500 万亿标记数据集，比其他公开预训练数据集产生更好性能的大型语言模型（LLM）。LLMs 在 FineWeb 上预训练表现出更好的知识和推理能力，我们公开释放了数据集、数据整理代码库和所有在消融实验中训练的模型。