The BigCode project is an open-scientific collaboration working on the
responsible development of large language models for code. This tech report
describes the progress of the collaboration until December 2022, outlining the
current state of the Personally Identifiable Information (PII) redaction
pipeline, the experiments conducted to de-risk the model architecture, and the
experiments investigating better preprocessing methods for the training data.
We train 1.1B parameter models on the Java, JavaScript, and Python subsets of
The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find
that more aggressive filtering of near-duplicates can further boost performance
and, surprisingly, that selecting files from repositories with 5+ GitHub stars
deteriorates performance significantly. Our best model outperforms previous
open-source multilingual code generation models (InCoder-6.7B and
CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java,
JavaScript, and Python portions of MultiPL-E, despite being a substantially
smaller model. All models are released under an OpenRAIL license at
this https URL.

本技术报告介绍了 BigCode 项目截至 2022 年 12 月的进展情况，包括当前状态的个人身份信息 (PII) 清理管道、减少模型架构风险的实验以及改进训练数据预处理方法的实验。我们在 The Stack 的 Java、JavaScript 和 Python 子集上训练了 11 亿参数模型，并在 MultiPL-E 的文本到代码基准测试上进行了评估。我们发现，更激进地过滤近似重复的数据可以进一步提高性能，并令人惊讶的是，从具有超过 5 个 GitHub 星的代码库中选择文件实际上会明显降低性能。我们最好的模型在 MultiPL-E 的 Java、JavaScript 和 Python 部分的从左到右生成和插值中都优于以前的开源多语言代码生成模型 (InCoder-6.7B 和 CodeGen-Multi-2.7B)，尽管它是一个相对较小的模型。所有模型均在 https://github.com/bigcode/BIGCODE 中以 OpenRAIL 许可证发布。