We introduce phi-1, a new large language model for code, with significantly
smaller size than competing models: phi-1 is a Transformer-based model with
1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook
quality" data from the web (6B tokens) and synthetically generated textbooks
and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays
surprising emergent properties compared to phi-1-base, our model before our
finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller
model with 350M parameters trained with the same pipeline as phi-1 that still
achieves 45% on HumanEval.

我们介绍了 Phi-1，这是一种新的大型代码语言模型，比现有的模型要小得多：Phi-1 是一种基于 Transformer 的模型，具有 13 亿个参数，使用来自 Web 的 “教科书品质” 数据（60 亿个标记）和使用 GPT-3.5 合成生成的教科书和练习进行了为期 4 天的训练。尽管规模较小，Phi-1 在 HumanEval 上的 pass@1 准确率为 50.6％，在 MBPP 上为 55.5％。与 phi-1-base（我们的模型在微调阶段之前）和具有相同工艺流程的 phi-1-small（具有 3.5 亿个参数的较小模型）相比，它还展示了令人惊讶的 emergent 属性，虽然仍然实现了 45％的 HumanEval。