The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

通过研究扩展定律，我们发现了DeepSeek LLM在两种常用的开源配置下，7B和67B，用于扩展大规模模型的独特发现，并介绍了DeepSeek LLM项目的长期前景。通过创建包含2万亿标记的数据集并不断扩展，来支持预训练阶段。我们在DeepSeek LLM基础模型上进行有监督的微调和直接偏好优化，从而创建了DeepSeek Chat模型。评估结果表明，DeepSeek LLM 67B在各种基准测试中均超过LLaMA-2 70B，尤其在代码、数学和推理领域。此外，开放式评估显示DeepSeek LLM 67B Chat在性能上优于GPT-3.5。

DeepSeek LLM: 以长期主义为基础扩展开源语言模型